Constraints for usage of updated motion information

ABSTRACT

Embodiments of the present disclosure relates to constraints for usage of updated motion information. A method for video processing is provided, including: determining original motion information associated with a current block; generating updated motion information based on specific prediction mode; and performing, based on the updated motion information, a conversion between the current block and a bitstream representation of a video data including the current block, wherein the specific prediction mode includes one or more of bi-directional optical flow (BIO) refinement, a decoder-side motion vector refinement (DMVR), frame-rate up conversion (FRUC) techniques or a template matching technique.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/IB2019/056642 filed on Aug. 5, 2019 which claims the priority toand benefits of International Patent Application No. PCT/CN2018/098691,filed on Aug. 4, 2018 and No. PCT/CN2018/109250, filed on Oct. 6, 2018.All the aforementioned patent applications are hereby incorporated byreference in their entireties.

TECHNICAL FIELD

This patent document relates to video coding techniques, devices andsystems.

BACKGROUND

In spite of the advances in video compression, digital video stillaccounts for the largest bandwidth use on the internet and other digitalcommunication networks. As the number of connected user devices capableof receiving and displaying video increases, it is expected that thebandwidth demand for digital video usage will continue to grow.

SUMMARY

Devices, systems and methods related to digital video coding, andspecifically, motion refinement based on updated motion vectors that aregenerated based on two-step inter-prediction are described. Thedescribed methods may be applied to both the existing video codingstandards (e.g., High Efficiency Video Coding (HEVC)) and future videocoding standards or video codecs.

In one representative aspect, a video processing method is provided,comprising: determining original motion information for a current block;scaling original motion vectors of the original motion information andderived motion vectors derived based on the original motion vectors to asame target precision; generating updated motion vectors from the scaledoriginal and derived motion vectors; and performing a conversion betweenthe current block and the bitstream representation of a video includingthe current block, based on the updated motion vectors.

In another representative aspect, a video processing method is provided,comprising: determining original motion information for a current block;updating original motion vectors of the original motion information forthe current block based on a refining method; clipping the updatedmotion vectors to be within a range; and performing a conversion betweenthe current block and a bitstream representation of a video includingthe current block, based on the clipped updated motion vectors.

In yet another representative aspect, the disclosed technology may beused to provide a method for video processing, comprising: determiningoriginal motion information associated with a current block; generatingupdated motion information based on specific prediction mode; andperforming, based on the updated motion information, a conversionbetween the current block and a bitstream representation of a video dataincluding the current block, wherein the specific prediction modeincludes one or more of bi-directional optical flow (BIO) refinement, adecoder-side motion vector refinement (DMVR), frame-rate up conversion(FRUC) techniques or a template matching technique.

In yet another representative aspect, a video processing method isprovided, comprising: determining a motion vector difference (MVD)precision for a current block processed with affine mode from an MVDprecision set; performing, based on the determined MVD precision, aconversion between the current block and a bitstream representation of avideo including the current block.

In yet another representative aspect, a video processing method isprovided, comprising: determining non-updated motion informationassociated with a current block; updating the non-updated motioninformation based on multiple decoder-side motion vector derivation(DMVD) methods to generate updated motion information for the currentblock; and performing, based on the updated motion information, aconversion between the current block and a bitstream representation of avideo including the current block.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesreceiving a bitstream representation of a current block of video data,generating updated first and second reference motion vectors based on aweighted sum of a first scaled motion vector and first and second scaledreference motion vectors, respectively, where a first motion vector isderived based on a first reference motion vector from a first referenceblock and a second reference motion vector from a second referenceblock, where the current block is associated with the first and secondreference blocks with the first scaled motion vector being generated byscaling the first motion vector to a target precision, and the first andsecond scaled reference motion vectors being generated by scaling thefirst and second reference motion vectors to the target precision,respectively, and processing the bitstream representation based on theupdated first and second reference motion vectors to generate thecurrent block.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesgenerating, for a current block, an intermediate prediction based on afirst motion information associated with the current block, updating thefirst motion information to a second motion information, and generatinga final prediction for the current block based on the intermediateprediction or the second motion information.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesreceiving a bitstream representation of a current block of video data,generating intermediate motion information based on motion informationassociated with the current block, generating updated first and secondreference motion vectors based on first and second reference motionvectors, respectively, where the current block is associated with firstand second reference blocks, and where the first and second referencemotion vectors are associated with the first and second referenceblocks, respectively, and processing the bitstream representation basedon the intermediate motion information or the updated first and secondreference motion vectors to generate the current block.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesgenerating, for a current block, an intermediate prediction based on afirst motion information associated with the current block, updating thefirst motion information to a second motion information, and generatinga final prediction for the current block based on the intermediateprediction or the second motion information.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesreceiving a bitstream representation of a current block of video data,generating intermediate motion information based on motion informationassociated with the current block, generating updated first and secondreference motion vectors based on first and second reference motionvectors, respectively, where the current block is associated with firstand second reference blocks, and where the first and second referencemotion vectors are associated with the first and second referenceblocks, respectively, and processing the bitstream representation basedon the intermediate motion information or the updated first and secondreference motion vectors to generate the current block.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesgenerating, for a current block, an intermediate prediction based on afirst motion information associated with the current block, updating thefirst motion information to a second motion information, and generatinga final prediction for the current block based on the intermediateprediction or the second motion information.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesreceiving a bitstream representation of a current block of video data,generating intermediate motion information based on motion informationassociated with the current block, generating updated first and secondreference motion vectors based on first and second reference motionvectors, respectively, where the current block is associated with firstand second reference blocks, and where the first and second referencemotion vectors are associated with the first and second referenceblocks, respectively, and processing the bitstream representation basedon the intermediate motion information or the updated first and secondreference motion vectors to generate the current block.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesgenerating, for a bitstream representation of a current block, anupdated reference block by modifying a reference block associated withthe current block, calculating, based on the updated reference block, atemporal gradient for a bi-directional optical flow (BIO) motionrefinement, and performing, based on the temporal gradient, aconversion, which includes the BIO motion refinement, between thebitstream representation and the current block.

In yet another representative aspect, the disclosed technology may beused to provide a method for video coding. This method includesgenerating, for a bitstream representation of a current block, atemporal gradient for a bi-directional optical flow (BIO) motionrefinement, generating an updated temporal gradient by subtracting adifference of a first mean value and a second mean value from thetemporal gradient, where the first mean value is a mean value for afirst reference block, the second mean value is a mean value for asecond reference block, and the first and second reference blocks areassociated with the current block, and performing, based on the updatedtemporal gradient, a conversion, which includes the BIO motionrefinement, between the bitstream representation and the current block.

In yet another representative aspect, the above-described method isembodied in the form of processor-executable code and stored in acomputer-readable program medium.

In yet another representative aspect, a device that is configured oroperable to perform the above-described method is disclosed. The devicemay include a processor that is programmed to implement this method.

In yet another representative aspect, a video decoder apparatus mayimplement a method as described herein.

The above and other aspects and features of the disclosed technology aredescribed in greater detail in the drawings, the description and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of constructing a merge candidate list.

FIG. 2 shows an example of positions of spatial candidates.

FIG. 3 shows an example of candidate pairs subject to a redundancy checkof spatial merge candidates.

FIGS. 4A and 4B show examples of the position of a second predictionunit (PU) based on the size and shape of the current block.

FIG. 5 shows an example of motion vector scaling for temporal mergecandidates.

FIG. 6 shows an example of candidate positions for temporal mergecandidates.

FIG. 7 shows an example of generating a combined bi-predictive mergecandidate.

FIG. 8 shows an example of constructing motion vector predictioncandidates.

FIG. 9 shows an example of motion vector scaling for spatial motionvector candidates.

FIG. 10 shows an example of motion prediction using the alternativetemporal motion vector prediction (ATMVP) algorithm for a coding unit(CU).

FIG. 11 shows an example of a coding unit (CU) with sub-blocks andneighboring blocks used by the spatial-temporal motion vector prediction(STMVP) algorithm.

FIGS. 12A and 12B show example snapshots of sub-block when using theoverlapped block motion compensation (OBMC) algorithm.

FIG. 13 shows an example of neighboring samples used to deriveparameters for the local illumination compensation (LIC) algorithm.

FIG. 14 shows an example of a simplified affine motion model.

FIG. 15 shows an example of an affine motion vector field (MVF) persub-block.

FIG. 16 shows an example of motion vector prediction (MVP) for theAF_INTER affine motion mode.

FIGS. 17A and 17B show example candidates for the AF_MERGE affine motionmode.

FIG. 18 shows an example of bilateral matching in pattern matched motionvector derivation (PMMVD) mode, which is a special merge mode based onthe frame-rate up conversion (FRUC) algorithm.

FIG. 19 shows an example of template matching in the FRUC algorithm.

FIG. 20 shows an example of unilateral motion estimation in the FRUCalgorithm.

FIG. 21 shows an example of an optical flow trajectory used by thebi-directional optical flow (BIO) algorithm.

FIGS. 22A and 22B show example snapshots of using of the bi-directionaloptical flow (BIO) algorithm without block extensions.

FIG. 23 shows an example of the decoder-side motion vector refinement(DMVR) algorithm based on bilateral template matching.

FIG. 24 shows an example of a template definition used in transformcoefficient context modeling.

FIG. 25 shows different examples of motion vector scaling.

FIGS. 26A and 26B show examples of inner and boundary sub-blocks in aPU/CU.

FIG. 27 shows a flowchart of an example method for video coding inaccordance with the presently disclosed technology.

FIG. 28 shows a flowchart of another example method for video coding inaccordance with the presently disclosed technology.

FIG. 29 shows a flowchart of another example method for video coding inaccordance with the presently disclosed technology.

FIG. 30 shows a flowchart of another example method for video coding inaccordance with the presently disclosed technology.

FIG. 31 shows a flowchart of another example method for video coding inaccordance with the presently disclosed technology.

FIG. 32 shows an example of deriving motion vector in bi-directionaloptical flow based video encoding.

FIG. 33 shows a flowchart of another example method for video coding inaccordance with the presently disclosed technology.

FIG. 34 shows a flowchart of another example method for video coding inaccordance with the presently disclosed technology.

FIG. 35 shows a flowchart of another example method for video coding inaccordance with the presently disclosed technology.

FIG. 36 shows a flowchart of another example method for video coding inaccordance with the presently disclosed technology.

FIG. 37 is a block diagram of an example of a hardware platform forimplementing a visual media decoding or a visual media encodingtechnique described in the present document.

FIG. 38 shows a flowchart of another example method for video processingin accordance with the presently disclosed technology.

FIG. 39 shows a flowchart of another example method for video processingin accordance with the presently disclosed technology.

FIG. 40 shows a flowchart of another example method for video processingin accordance with the presently disclosed technology.

FIG. 41 shows a flowchart of another example method for video processingin accordance with the presently disclosed technology.

FIG. 42 shows a flowchart of another example method for video processingin accordance with the presently disclosed technology.

DETAILED DESCRIPTION

Due to the increasing demand of higher resolution video, video codingmethods and techniques are ubiquitous in modern technology. Video codecstypically include an electronic circuit or software that compresses ordecompresses digital video, and are continually being improved toprovide higher coding efficiency. A video codec converts uncompressedvideo to a compressed format or vice versa. There are complexrelationships between the video quality, the amount of data used torepresent the video (determined by the bit rate), the complexity of theencoding and decoding algorithms, sensitivity to data losses and errors,ease of editing, random access, and end-to-end delay (latency). Thecompressed format usually conforms to a standard video compressionspecification, e.g., the High Efficiency Video Coding (HEVC) standard(also known as H.265 or MPEG-H Part 2), the Versatile Video Codingstandard to be finalized, or other current and/or future video codingstandards.

Embodiments of the disclosed technology may be applied to existing videocoding standards (e.g., HEVC, H.265) and future standards to improvecompression performance. Section headings are used in the presentdocument to improve readability of the description and do not in any waylimit the discussion or the embodiments (and/or implementations) to therespective sections only.

1. Examples of Inter-Prediction in HEVC/H.265

Video coding standards have significantly improved over the years, andnow provide, in part, high coding efficiency and support for higherresolutions. Recent standards such as HEVC and H.265 are based on thehybrid video coding structure wherein temporal prediction plus transformcoding are utilized.

1.1 Examples of Prediction Modes

Each inter-predicted PU (prediction unit) has motion parameters for oneor two reference picture lists. In some embodiments, motion parametersinclude a motion vector and a reference picture index. In otherembodiments, the usage of one of the two reference picture lists mayalso be signaled using inter_pred_idc. In yet other embodiments, motionvectors may be explicitly coded as deltas relative to predictors.

When a CU is coded with skip mode, one PU is associated with the CU, andthere are no significant residual coefficients, no coded motion vectordelta or reference picture index. A merge mode is specified whereby themotion parameters for the current PU are obtained from neighboring PUs,including spatial and temporal candidates. The merge mode can be appliedto any inter-predicted PU, not only for skip mode. The alternative tomerge mode is the explicit transmission of motion parameters, wheremotion vector, corresponding reference picture index for each referencepicture list and reference picture list usage are signaled explicitlyper each PU.

When signaling indicates that one of the two reference picture lists isto be used, the PU is produced from one block of samples. This isreferred to as ‘uni-prediction’. Uni-prediction is available both forP-slices and B-slices.

When signaling indicates that both of the reference picture lists are tobe used, the PU is produced from two blocks of samples. This is referredto as ‘bi-prediction’. Bi-prediction is available for B-slices only.

1.1.1 Embodiments of Constructing Candidates for Merge Mode

When a PU is predicted using merge mode, an index pointing to an entryin the merge candidates list is parsed from the bitstream and used toretrieve the motion information. The construction of this list can besummarized according to the following sequence of steps:

Step 1: Initial candidates derivation

-   -   Step 1.1: Spatial candidates derivation    -   Step 1.2: Redundancy check for spatial candidates    -   Step 1.3: Temporal candidates derivation

Step 2: Additional candidates insertion

-   -   Step 2.1: Creation of bi-predictive candidates    -   Step 2.2: Insertion of zero motion candidates

FIG. 1 shows an example of constructing a merge candidate list based onthe sequence of steps summarized above. For spatial merge candidatederivation, a maximum of four merge candidates are selected amongcandidates that are located in five different positions. For temporalmerge candidate derivation, a maximum of one merge candidate is selectedamong two candidates. Since constant number of candidates for each PU isassumed at decoder, additional candidates are generated when the numberof candidates does not reach to maximum number of merge candidate(MaxNumMergeCand) which is signaled in slice header. Since the number ofcandidates is constant, index of best merge candidate is encoded usingtruncated unary binarization (TU). If the size of CU is equal to 8, allthe PUs of the current CU share a single merge candidate list, which isidentical to the merge candidate list of the 2N×2N prediction unit.

1.1.2 Constructing Spatial Merge Candidates

In the derivation of spatial merge candidates, a maximum of four mergecandidates are selected among candidates located in the positionsdepicted in FIG. 2. The order of derivation is A₁, B₁, B₀, A₀ and B₂.Position B₂ is considered only when any PU of position A₁, B₁, B₀, A₀ isnot available (e.g. because it belongs to another slice or tile) or isintra coded. After candidate at position A₁ is added, the addition ofthe remaining candidates is subject to a redundancy check which ensuresthat candidates with same motion information are excluded from the listso that coding efficiency is improved.

To reduce computational complexity, not all possible candidate pairs areconsidered in the mentioned redundancy check. Instead only the pairslinked with an arrow in FIG. 3 are considered and a candidate is onlyadded to the list if the corresponding candidate used for redundancycheck has not the same motion information. Another source of duplicatemotion information is the “second PU” associated with partitionsdifferent from 2N×2N. As an example, FIGS. 4A and 4B depict the secondPU for the case of N×2N and 2N×N, respectively. When the current PU ispartitioned as N×2N, candidate at position A₁ is not considered for listconstruction. In some embodiments, adding this candidate may lead to twoprediction units having the same motion information, which is redundantto just have one PU in a coding unit. Similarly, position B₁ is notconsidered when the current PU is partitioned as 2N×N.

1.1.3 Constructing Temporal Merge Candidates

In this step, only one candidate is added to the list. Particularly, inthe derivation of this temporal merge candidate, a scaled motion vectoris derived based on co-located PU belonging to the picture which has thesmallest POC difference with current picture within the given referencepicture list. The reference picture list to be used for derivation ofthe co-located PU is explicitly signaled in the slice header.

FIG. 5 shows an example of the derivation of the scaled motion vectorfor a temporal merge candidate (as the dotted line), which is scaledfrom the motion vector of the co-located PU using the POC distances, tband td, where tb is defined to be the POC difference between thereference picture of the current picture and the current picture and tdis defined to be the POC difference between the reference picture of theco-located picture and the co-located picture. The reference pictureindex of temporal merge candidate is set equal to zero. For a B-slice,two motion vectors, one is for reference picture list 0 and the other isfor reference picture list 1, are obtained and combined to make thebi-predictive merge candidate.

In the co-located PU (Y) belonging to the reference frame, the positionfor the temporal candidate is selected between candidates C₀ and C₁, asdepicted in FIG. 6. If PU at position C₀ is not available, is intracoded, or is outside of the current CTU, position C₁ is used. Otherwise,position C₀ is used in the derivation of the temporal merge candidate.

1.1.4 Constructing Additional Types of Merge Candidates

Besides spatio-temporal merge candidates, there are two additional typesof merge candidates: combined bi-predictive merge candidate and zeromerge candidate. Combined bi-predictive merge candidates are generatedby utilizing spatio-temporal merge candidates. Combined bi-predictivemerge candidate is used for B-Slice only. The combined bi-predictivecandidates are generated by combining the first reference picture listmotion parameters of an initial candidate with the second referencepicture list motion parameters of another. If these two tuples providedifferent motion hypotheses, they will form a new bi-predictivecandidate.

FIG. 7 shows an example of this process, wherein two candidates in theoriginal list (710, on the left), which have mvL0 and refIdxL0 or mvL1and refIdxL1, are used to create a combined bi-predictive mergecandidate added to the final list (720, on the right).

Zero motion candidates are inserted to fill the remaining entries in themerge candidates list and therefore hit the MaxNumMergeCand capacity.These candidates have zero spatial displacement and a reference pictureindex which starts from zero and increases every time a new zero motioncandidate is added to the list. The number of reference frames used bythese candidates is one and two for uni- and bi-directional prediction,respectively. In some embodiments, no redundancy check is performed onthese candidates.

1.1.5 Examples of Motion Estimation Regions for Parallel Processing

To speed up the encoding process, motion estimation can be performed inparallel whereby the motion vectors for all prediction units inside agiven region are derived simultaneously. The derivation of mergecandidates from spatial neighborhood may interfere with parallelprocessing as one prediction unit cannot derive the motion parametersfrom an adjacent PU until its associated motion estimation is completed.To mitigate the trade-off between coding efficiency and processinglatency, a motion estimation region (MER) may be defined. The size ofthe MER may be signaled in the picture parameter set (PPS) using the“log 2_parallel_merge_level_minus2” syntax element. When a MER isdefined, merge candidates falling in the same region are marked asunavailable and therefore not considered in the list construction.

1.2 Embodiments of Advanced Motion Vector Prediction (AMVP)

AMVP exploits spatio-temporal correlation of motion vector withneighboring PUs, which is used for explicit transmission of motionparameters. It constructs a motion vector candidate list by firstlychecking availability of left, above temporally neighboring PUpositions, removing redundant candidates and adding zero vector to makethe candidate list to be constant length. Then, the encoder can selectthe best predictor from the candidate list and transmit thecorresponding index indicating the chosen candidate. Similarly withmerge index signaling, the index of the best motion vector candidate isencoded using truncated unary. The maximum value to be encoded in thiscase is 2 (see FIG. 8). In the following sections, details aboutderivation process of motion vector prediction candidate are provided.

1.2.1 Examples of Constructing Motion Vector Prediction Candidates

FIG. 8 summarizes derivation process for motion vector predictioncandidate, and may be implemented for each reference picture list withrefidx as an input.

In motion vector prediction, two types of motion vector candidates areconsidered: spatial motion vector candidate and temporal motion vectorcandidate. For spatial motion vector candidate derivation, two motionvector candidates are eventually derived based on motion vectors of eachPU located in five different positions as previously shown in FIG. 2.

For temporal motion vector candidate derivation, one motion vectorcandidate is selected from two candidates, which are derived based ontwo different co-located positions. After the first list ofspatio-temporal candidates is made, duplicated motion vector candidatesin the list are removed. If the number of potential candidates is largerthan two, motion vector candidates whose reference picture index withinthe associated reference picture list is larger than 1 are removed fromthe list. If the number of spatio-temporal motion vector candidates issmaller than two, additional zero motion vector candidates is added tothe list.

1.2.2 Constructing Spatial Motion Vector Candidates

In the derivation of spatial motion vector candidates, a maximum of twocandidates are considered among five potential candidates, which arederived from PUs located in positions as previously shown in FIG. 2,those positions being the same as those of motion merge. The order ofderivation for the left side of the current PU is defined as A₀, A₁, andscaled A₀, scaled A₁. The order of derivation for the above side of thecurrent PU is defined as B₀, B₁, B₂, scaled B₀, scaled B₁, scaled B₂.For each side there are therefore four cases that can be used as motionvector candidate, with two cases not required to use spatial scaling,and two cases where spatial scaling is used. The four different casesare summarized as follows:

-   -   No spatial scaling        -   (1) Same reference picture list, and same reference picture            index (same POC)        -   (2) Different reference picture list, but same reference            picture (same POC)    -   Spatial scaling        -   (3) Same reference picture list, but different reference            picture (different POC)        -   (4) Different reference picture list, and different            reference picture (different POC)

The no-spatial-scaling cases are checked first followed by the casesthat allow spatial scaling. Spatial scaling is considered when the POCis different between the reference picture of the neighbouring PU andthat of the current PU regardless of reference picture list. If all PUsof left candidates are not available or are intra coded, scaling for theabove motion vector is allowed to help parallel derivation of left andabove MV candidates. Otherwise, spatial scaling is not allowed for theabove motion vector.

As shown in the example in FIG. 9, for the spatial scaling case, themotion vector of the neighbouring PU is scaled in a similar manner asfor temporal scaling. One difference is that the reference picture listand index of current PU is given as input; the actual scaling process isthe same as that of temporal scaling.

1.2.3 Constructing Temporal Motion Vector Candidates

Apart from the reference picture index derivation, all processes for thederivation of temporal merge candidates are the same as for thederivation of spatial motion vector candidates (as shown in the examplein FIG. 6). In some embodiments, the reference picture index is signaledto the decoder.

2. Example of Inter Prediction Methods in Joint Exploration Model (JEM)

In some embodiments, future video coding technologies are explored usinga reference software known as the Joint Exploration Model (JEM). In JEM,sub-block based prediction is adopted in several coding tools, such asaffine prediction, alternative temporal motion vector prediction(ATMVP), spatial-temporal motion vector prediction (STMVP),bi-directional optical flow (BIO), Frame-Rate Up Conversion (FRUC),Locally Adaptive Motion Vector Resolution (LAMVR), Overlapped BlockMotion Compensation (OBMC), Local Illumination Compensation (LIC), andDecoder-side Motion Vector Refinement (DMVR).

2.1 Examples of Sub-CU Based Motion Vector Prediction

In the JEM with quadtrees plus binary trees (QTBT), each CU can have atmost one set of motion parameters for each prediction direction. In someembodiments, two sub-CU level motion vector prediction methods areconsidered in the encoder by splitting a large CU into sub-CUs andderiving motion information for all the sub-CUs of the large CU.Alternative temporal motion vector prediction (ATMVP) method allows eachCU to fetch multiple sets of motion information from multiple blockssmaller than the current CU in the collocated reference picture. Inspatial-temporal motion vector prediction (STMVP) method motion vectorsof the sub-CUs are derived recursively by using the temporal motionvector predictor and spatial neighbouring motion vector. In someembodiments, and to preserve more accurate motion field for sub-CUmotion prediction, the motion compression for the reference frames maybe disabled.

2.1.1 Examples of Alternative Temporal Motion Vector Prediction (ATMVP)

In the ATMVP method, the temporal motion vector prediction (TMVP) methodis modified by fetching multiple sets of motion information (includingmotion vectors and reference indices) from blocks smaller than thecurrent CU.

FIG. 10 shows an example of ATMVP motion prediction process for a CU1000. The ATMVP method predicts the motion vectors of the sub-CUs 1001within a CU 1000 in two steps. The first step is to identify thecorresponding block 1051 in a reference picture 1050 with a temporalvector. The reference picture 1050 is also referred to as the motionsource picture. The second step is to split the current CU 1000 intosub-CUs 1001 and obtain the motion vectors as well as the referenceindices of each sub-CU from the block corresponding to each sub-CU.

In the first step, a reference picture 1050 and the corresponding blockis determined by the motion information of the spatial neighboringblocks of the current CU 1000. To avoid the repetitive scanning processof neighboring blocks, the first merge candidate in the merge candidatelist of the current CU 1000 is used. The first available motion vectoras well as its associated reference index are set to be the temporalvector and the index to the motion source picture. This way, thecorresponding block may be more accurately identified, compared withTMVP, wherein the corresponding block (sometimes called collocatedblock) is always in a bottom-right or center position relative to thecurrent CU.

In the second step, a corresponding block of the sub-CU 1051 isidentified by the temporal vector in the motion source picture 1050, byadding to the coordinate of the current CU the temporal vector. For eachsub-CU, the motion information of its corresponding block (e.g., thesmallest motion grid that covers the center sample) is used to derivethe motion information for the sub-CU. After the motion information of acorresponding N×N block is identified, it is converted to the motionvectors and reference indices of the current sub-CU, in the same way asTMVP of HEVC, wherein motion scaling and other procedures apply. Forexample, the decoder checks whether the low-delay condition (e.g. thePOCs of all reference pictures of the current picture are smaller thanthe POC of the current picture) is fulfilled and possibly uses motionvector MVx (e.g., the motion vector corresponding to reference picturelist X) to predict motion vector MVy (e.g., with X being equal to 0 or 1and Y being equal to 1−X) for each sub-CU.

2.1.2 Examples of Spatial-Temporal Motion Vector Prediction (STMVP)

In the STMVP method, the motion vectors of the sub-CUs are derivedrecursively, following raster scan order. FIG. 11 shows an example ofone CU with four sub-blocks and neighboring blocks. Consider an 8×8 CU1100 that includes four 4×4 sub-CUs A (1101), B (1102), C (1103), and D(1104). The neighboring 4×4 blocks in the current frame are labeled as a(1111), b (1112), c (1113), and d (1114).

The motion derivation for sub-CU A starts by identifying its two spatialneighbors. The first neighbor is the N×N block above sub-CU A 1101(block c 1113). If this block c (1113) is not available or is intracoded the other N×N blocks above sub-CU A (1101) are checked (from leftto right, starting at block c 1113). The second neighbor is a block tothe left of the sub-CU A 1101 (block b 1112). If block b (1112) is notavailable or is intra coded other blocks to the left of sub-CU A 1101are checked (from top to bottom, staring at block b 1112). The motioninformation obtained from the neighboring blocks for each list is scaledto the first reference frame for a given list. Next, temporal motionvector predictor (TMVP) of sub-block A 1101 is derived by following thesame procedure of TMVP derivation as specified in HEVC. The motioninformation of the collocated block at block D 1104 is fetched andscaled accordingly. Finally, after retrieving and scaling the motioninformation, all available motion vectors are averaged separately foreach reference list. The averaged motion vector is assigned as themotion vector of the current sub-CU.

2.1.3 Examples of Sub-CU Motion Prediction Mode Signaling

In some embodiments, the sub-CU modes are enabled as additional mergecandidates and there is no additional syntax element required to signalthe modes. Two additional merge candidates are added to merge candidateslist of each CU to represent the ATMVP mode and STMVP mode. In otherembodiments, up to seven merge candidates may be used, if the sequenceparameter set indicates that ATMVP and STMVP are enabled. The encodinglogic of the additional merge candidates is the same as for the mergecandidates in the HM, which means, for each CU in P or B slice, two moreRD checks may be needed for the two additional merge candidates. In someembodiments, e.g., JEM, all bins of the merge index are context coded byCABAC (Context-based Adaptive Binary Arithmetic Coding). In otherembodiments, e.g., HEVC, only the first bin is context coded and theremaining bins are context by-pass coded.

2.2 Examples of Adaptive Motion Vector Difference Resolution

In some embodiments, motion vector differences (MVDs) (between themotion vector and predicted motion vector of a PU) are signaled in unitsof quarter luma samples when use_integer_mv_flag is equal to 0 in theslice header. In the JEM, a locally adaptive motion vector resolution(LAMVR) is introduced. In the JEM, MVD can be coded in units of quarterluma samples, integer luma samples or four luma samples. The MVDresolution is controlled at the coding unit (CU) level, and MVDresolution flags are conditionally signaled for each CU that has atleast one non-zero MVD components.

For a CU that has at least one non-zero MVD components, a first flag issignaled to indicate whether quarter luma sample MV precision is used inthe CU. When the first flag (equal to 1) indicates that quarter lumasample MV precision is not used, another flag is signaled to indicatewhether integer luma sample MV precision or four luma sample MVprecision is used.

When the first MVD resolution flag of a CU is zero, or not coded for aCU (meaning all MVDs in the CU are zero), the quarter luma sample MVresolution is used for the CU. When a CU uses integer-luma sample MVprecision or four-luma-sample MV precision, the MVPs in the AMVPcandidate list for the CU are rounded to the corresponding precision.

In the encoder, CU-level RD checks are used to determine which MVDresolution is to be used for a CU. That is, the CU-level RD check isperformed three times for each MVD resolution. To accelerate encoderspeed, the following encoding schemes are applied in the JEM:

-   -   During RD check of a CU with normal quarter luma sample MVD        resolution, the motion information of the current CU (integer        luma sample accuracy) is stored. The stored motion information        (after rounding) is used as the starting point for further small        range motion vector refinement during the RD check for the same        CU with integer luma sample and 4 luma sample MVD resolution so        that the time-consuming motion estimation process is not        duplicated three times.    -   RD check of a CU with 4 luma sample MVD resolution is        conditionally invoked. For a CU, when RD cost integer luma        sample MVD resolution is much larger than that of quarter luma        sample MVD resolution, the RD check of 4 luma sample MVD        resolution for the CU is skipped.

2.3 Examples of Higher Motion Vector Storage Accuracy

In HEVC, motion vector accuracy is one-quarter pel (one-quarter lumasample and one-eighth chroma sample for 4:2:0 video). In the JEM, theaccuracy for the internal motion vector storage and the merge candidateincreases to 1/16 pel. The higher motion vector accuracy ( 1/16 pel) isused in motion compensation inter prediction for the CU coded withskip/merge mode. For the CU coded with normal AMVP mode, either theinteger-pel or quarter-pel motion is used.

SHVC upsampling interpolation filters, which have same filter length andnormalization factor as HEVC motion compensation interpolation filters,are used as motion compensation interpolation filters for the additionalfractional pel positions. The chroma component motion vector accuracy is1/32 sample in the JEM, the additional interpolation filters of 1/32 pelfractional positions are derived by using the average of the filters ofthe two neighbouring 1/16 pel fractional positions.

2.4 Examples of Overlapped Block Motion Compensation (OBMC)

In the JEM, OBMC can be switched on and off using syntax at the CUlevel. When OBMC is used in the JEM, the OBMC is performed for allmotion compensation (MC) block boundaries except the right and bottomboundaries of a CU. Moreover, it is applied for both the luma and chromacomponents. In the JEM, an MC block corresponds to a coding block. Whena CU is coded with sub-CU mode (includes sub-CU merge, affine and FRUCmode), each sub-block of the CU is a MC block. To process CU boundariesin a uniform fashion, OBMC is performed at sub-block level for all MCblock boundaries, where sub-block size is set equal to 4×4, as shown inFIGS. 12A and 12B.

FIG. 12A shows sub-blocks at the CU/PU boundary, and the hatchedsub-blocks are where OBMC applies. Similarly, FIG. 12B shows the sub-Pusin ATMVP mode.

When OBMC applies to the current sub-block, besides current motionvectors, motion vectors of four connected neighboring sub-blocks, ifavailable and are not identical to the current motion vector, are alsoused to derive prediction block for the current sub-block. Thesemultiple prediction blocks based on multiple motion vectors are combinedto generate the final prediction signal of the current sub-block.

Prediction block based on motion vectors of a neighboring sub-block isdenoted as PN, with N indicating an index for the neighboring above,below, left and right sub-blocks and prediction block based on motionvectors of the current sub-block is denoted as PC. When PN is based onthe motion information of a neighboring sub-block that contains the samemotion information to the current sub-block, the OBMC is not performedfrom PN. Otherwise, every sample of PN is added to the same sample inPC, i.e., four rows/columns of PN are added to PC. The weighting factors{¼, ⅛, 1/16, 1/32} are used for PN and the weighting factors {¾, ⅞,15/16, 31/32} are used for PC. The exception are small MC blocks, (i.e.,when height or width of the coding block is equal to 4 or a CU is codedwith sub-CU mode), for which only two rows/columns of PN are added toPC. In this case weighting factors {¼, ⅛} are used for PN and weightingfactors {¾, ⅞} are used for PC. For PN generated based on motion vectorsof vertically (horizontally) neighboring sub-block, samples in the samerow (column) of PN are added to PC with a same weighting factor.

In the JEM, for a CU with size less than or equal to 256 luma samples, aCU level flag is signaled to indicate whether OBMC is applied or not forthe current CU. For the CUs with size larger than 256 luma samples ornot coded with AMVP mode, OBMC is applied by default. At the encoder,when OBMC is applied for a CU, its impact is taken into account duringthe motion estimation stage. The prediction signal formed by OBMC usingmotion information of the top neighboring block and the left neighboringblock is used to compensate the top and left boundaries of the originalsignal of the current CU, and then the normal motion estimation processis applied.

2.5 Examples of Local Illumination Compensation (LIC)

LIC is based on a linear model for illumination changes, using a scalingfactor a and an offset b. And it is enabled or disabled adaptively foreach inter-mode coded coding unit (CU).

When LIC applies for a CU, a least square error method is employed toderive the parameters a and b by using the neighboring samples of thecurrent CU and their corresponding reference samples. FIG. 13 shows anexample of neighboring samples used to derive parameters of the ICalgorithm. Specifically, and as shown in FIG. 13, the subsampled (2:1subsampling) neighbouring samples of the CU and the correspondingsamples (identified by motion information of the current CU or sub-CU)in the reference picture are used. The IC parameters are derived andapplied for each prediction direction separately.

When a CU is coded with merge mode, the LIC flag is copied fromneighboring blocks, in a way similar to motion information copy in mergemode; otherwise, an LIC flag is signaled for the CU to indicate whetherLIC applies or not.

When LIC is enabled for a picture, an additional CU level RD check isneeded to determine whether LIC is applied or not for a CU. When LIC isenabled for a CU, the mean-removed sum of absolute difference (MR-SAD)and mean-removed sum of absolute Hadamard-transformed difference(MR-SATD) are used, instead of SAD and SATD, for integer pel motionsearch and fractional pel motion search, respectively.

To reduce the encoding complexity, the following encoding scheme isapplied in the JEM:

-   -   LIC is disabled for the entire picture when there is no obvious        illumination change between a current picture and its reference        pictures. To identify this situation, histograms of a current        picture and every reference picture of the current picture are        calculated at the encoder. If the histogram difference between        the current picture and every reference picture of the current        picture is smaller than a given threshold, LIC is disabled for        the current picture; otherwise, LIC is enabled for the current        picture.

2.6 Examples of Affine Motion Compensation Prediction

In HEVC, only a translation motion model is applied for motioncompensation prediction (MCP). However, the camera and objects may havemany kinds of motion, e.g. zoom in/out, rotation, perspective motions,and/or other irregular motions. JEM, on the other hand, applies asimplified affine transform motion compensation prediction. FIG. 14shows an example of an affine motion field of a block 1400 described bytwo control point motion vectors V₀ and V₁. The motion vector field(MVF) of the block 1400 can be described by the following equation:

$\begin{matrix}\left\{ \begin{matrix}{v_{x} = {{\frac{\left( {v_{1x} - v_{0x}} \right)}{w}x} - {\frac{\left( {v_{1y} - v_{0y}} \right)}{w}y} + v_{0x}}} \\{v_{y} = {{\frac{\left( {v_{1y} - v_{0y}} \right)}{w}x} + {\frac{\left( {v_{1x} - v_{0x}} \right)}{w}y} + v_{0y}}}\end{matrix} \right. & {{Eq}.\mspace{14mu} (1)}\end{matrix}$

As shown in FIG. 14, (v_(0x), v_(0y)) is motion vector of the top-leftcorner control point, and (v_(1x), v_(1y)) is motion vector of thetop-right corner control point. To simplify the motion compensationprediction, sub-block based affine transform prediction can be applied.The sub-block size M×N is derived as follows:

$\begin{matrix}\left\{ \begin{matrix}{M = {{clip}\; 3\left( {4,w,\frac{w \times {MvPre}}{\max \left( {{{abs}\left( {v_{1x} - v_{0x}} \right)},{{abs}\left( {v_{1y} - v_{0y}} \right)}} \right)}} \right)}} \\{N = {{clip}\; 3\left( {4,h,\frac{h \times {MvPre}}{\max \left( {{{abs}\left( {v_{2x} - v_{0x}} \right)},{{abs}\left( {v_{2y} - v_{0y}} \right)}} \right)}} \right)}}\end{matrix} \right. & {{Eq}.\mspace{14mu} (2)}\end{matrix}$

Here, MvPre is the motion vector fraction accuracy (e.g., 1/16 in JEM).(v_(2x), v_(2y)) is motion vector of the bottom-left control point,calculated according to Eq. (1). M and N can be adjusted downward ifnecessary to make it a divisor of w and h, respectively.

FIG. 15 shows an example of affine MVF per sub-block for a block 1500.To derive motion vector of each M×N sub-block, the motion vector of thecenter sample of each sub-block can be calculated according to Eq. (1),and rounded to the motion vector fraction accuracy (e.g., 1/16 in JEM).Then the motion compensation interpolation filters can be applied togenerate the prediction of each sub-block with derived motion vector.After the MCP, the high accuracy motion vector of each sub-block isrounded and saved as the same accuracy as the normal motion vector.

In the JEM, there are two affine motion modes: AF_INTER mode andAF_MERGE mode. For CUs with both width and height larger than 8,AF_INTER mode can be applied. An affine flag in CU level is signaled inthe bitstream to indicate whether AF_INTER mode is used. In the AF_INTERmode, a candidate list with motion vector pair {(v₀, v₁)|v₀={v_(A),v_(B), v_(c)}, v₁={v_(D), v_(E)}} is constructed using the neighboringblocks.

FIG. 16 shows an example of motion vector prediction (MVP) for a block1600 in the AF_INTER mode. As shown in FIG. 16, v₀ is selected from themotion vectors of the sub-block A, B, or C. The motion vectors from theneighboring blocks can be scaled according to the reference list. Themotion vectors can also be scaled according to the relationship amongthe Picture Order Count (POC) of the reference for the neighboringblock, the POC of the reference for the current CU, and the POC of thecurrent CU. The approach to select v₁ from the neighboring sub-block Dand E is similar. If the number of candidate list is smaller than 2, thelist is padded by the motion vector pair composed by duplicating each ofthe AMVP candidates. When the candidate list is larger than 2, thecandidates can be firstly sorted according to the neighboring motionvectors (e.g., based on the similarity of the two motion vectors in apair candidate). In some implementations, the first two candidates arekept. In some embodiments, a Rate Distortion (RD) cost check is used todetermine which motion vector pair candidate is selected as the controlpoint motion vector prediction (CPMVP) of the current CU. An indexindicating the position of the CPMVP in the candidate list can besignaled in the bitstream. After the CPMVP of the current affine CU isdetermined, affine motion estimation is applied and the control pointmotion vector (CPMV) is found. Then the difference of the CPMV and theCPMVP is signaled in the bitstream.

When a CU is applied in AF_MERGE mode, it gets the first block codedwith an affine mode from the valid neighboring reconstructed blocks.FIG. 17A shows an example of the selection order of candidate blocks fora current CU 1700. As shown in FIG. 17A, the selection order can be fromleft (1701), above (1702), above right (1703), left bottom (1704) toabove left (1705) of the current CU 1700. FIG. 17B shows another exampleof candidate blocks for a current CU 1700 in the AF_MERGE mode. If theneighboring left bottom block 1701 is coded in affine mode, as shown inFIG. 17B, the motion vectors v₂, v₃ and v₄ of the top left corner, aboveright corner, and left bottom corner of the CU containing the sub-block1701 are derived. The motion vector v₀ of the top left corner on thecurrent CU 1700 is calculated based on v2, v3 and v4. The motion vectorv1 of the above right of the current CU can be calculated accordingly.

After the CPMV of the current CU v0 and v1 are computed according to theaffine motion model in Eq. (1), the MVF of the current CU can begenerated. In order to identify whether the current CU is coded withAF_MERGE mode, an affine flag can be signaled in the bitstream whenthere is at least one neighboring block is coded in affine mode.

2.7 Examples of Pattern Matched Motion Vector Derivation (PMMVD)

The PMMVD mode is a special merge mode based on the Frame-Rate UpConversion (FRUC) method. With this mode, motion information of a blockis not signaled but derived at decoder side.

A FRUC flag can be signaled for a CU when its merge flag is true. Whenthe FRUC flag is false, a merge index can be signaled and the regularmerge mode is used. When the FRUC flag is true, an additional FRUC modeflag can be signaled to indicate which method (e.g., bilateral matchingor template matching) is to be used to derive motion information for theblock.

At the encoder side, the decision on whether using FRUC merge mode for aCU is based on RD cost selection as done for normal merge candidate. Forexample, multiple matching modes (e.g., bilateral matching and templatematching) are checked for a CU by using RD cost selection. The oneleading to the minimal cost is further compared to other CU modes. If aFRUC matching mode is the most efficient one, FRUC flag is set to truefor the CU and the related matching mode is used.

Typically, motion derivation process in FRUC merge mode has two steps: aCU-level motion search is first performed, then followed by a Sub-CUlevel motion refinement. At CU level, an initial motion vector isderived for the whole CU based on bilateral matching or templatematching. First, a list of MV candidates is generated and the candidatethat leads to the minimum matching cost is selected as the startingpoint for further CU level refinement. Then a local search based onbilateral matching or template matching around the starting point isperformed. The MV results in the minimum matching cost is taken as theMV for the whole CU. Subsequently, the motion information is furtherrefined at sub-CU level with the derived CU motion vectors as thestarting points.

For example, the following derivation process is performed for a W×H CUmotion information derivation. At the first stage, MV for the whole W×HCU is derived. At the second stage, the CU is further split into M×Msub-CUs. The value of M is calculated as in Eq. (3), D is a predefinedsplitting depth which is set to 3 by default in the JEM. Then the MV foreach sub-CU is derived.

$\begin{matrix}{M = {\max \left\{ {4,{\min \left\{ {\frac{M}{2^{D}},\frac{N}{2^{D}}} \right\}}} \right\}}} & {{Eq}.\mspace{14mu} (3)}\end{matrix}$

FIG. 18 shows an example of bilateral matching used in the Frame-Rate UpConversion (FRUC) method. The bilateral matching is used to derivemotion information of the current CU by finding the closest matchbetween two blocks along the motion trajectory of the current CU (1800)in two different reference pictures (1810, 1811). Under the assumptionof continuous motion trajectory, the motion vectors MV0 (1801) and MV1(1802) pointing to the two reference blocks are proportional to thetemporal distances, e.g., TD0 (1803) and TD1 (1804), between the currentpicture and the two reference pictures. In some embodiments, when thecurrent picture 1800 is temporally between the two reference pictures(1810, 1811) and the temporal distance from the current picture to thetwo reference pictures is the same, the bilateral matching becomesmirror based bi-directional MV.

FIG. 19 shows an example of template matching used in the Frame-Rate UpConversion (FRUC) method. Template matching can be used to derive motioninformation of the current CU 1900 by finding the closest match betweena template (e.g., top and/or left neighboring blocks of the current CU)in the current picture and a block (e.g., same size to the template) ina reference picture 1910. Except the aforementioned FRUC merge mode, thetemplate matching can also be applied to AMVP mode. In both JEM andHEVC, AMVP has two candidates. With the template matching method, a newcandidate can be derived. If the newly derived candidate by templatematching is different to the first existing AMVP candidate, it isinserted at the very beginning of the AMVP candidate list and then thelist size is set to two (e.g., by removing the second existing AMVPcandidate). When applied to AMVP mode, only CU level search is applied.

The MV candidate set at CU level can include the following: (1) originalAMVP candidates if the current CU is in AMVP mode, (2) all mergecandidates, (3) several MVs in the interpolated MV field (describedlater), and top and left neighboring motion vectors.

When using bilateral matching, each valid MV of a merge candidate can beused as an input to generate a MV pair with the assumption of bilateralmatching. For example, one valid MV of a merge candidate is (MVa,ref_(a)) at reference list A. Then the reference picture ref_(b) of itspaired bilateral MV is found in the other reference list B so thatref_(a) and ref_(b) are temporally at different sides of the currentpicture. If such a ref_(b) is not available in reference list B, ref_(b)is determined as a reference which is different from ref_(a) and itstemporal distance to the current picture is the minimal one in list B.After ref_(b) is determined, MVb is derived by scaling MVa based on thetemporal distance between the current picture and ref_(a), ref_(b).

In some implementations, four MVs from the interpolated MV field canalso be added to the CU level candidate list. More specifically, theinterpolated MVs at the position (0, 0), (W/2, 0), (0, H/2) and (W/2,H/2) of the current CU are added. When FRUC is applied in AMVP mode, theoriginal AMVP candidates are also added to CU level MV candidate set. Insome implementations, at the CU level, 15 MVs for AMVP CUs and 13 MVsfor merge CUs can be added to the candidate list.

The MV candidate set at sub-CU level includes an MV determined from aCU-level search, (2) top, left, top-left and top-right neighboring MVs,(3) scaled versions of collocated MVs from reference pictures, (4) oneor more ATMVP candidates (e.g., up to four), and (5) one or more STMVPcandidates (e.g., up to four). The scaled MVs from reference picturesare derived as follows. The reference pictures in both lists aretraversed. The MVs at a collocated position of the sub-CU in a referencepicture are scaled to the reference of the starting CU-level MV. ATMVPand STMVP candidates can be the four first ones. At the sub-CU level,one or more MVs (e.g., up to 17) are added to the candidate list.

Generation of an interpolated MV field. Before coding a frame,interpolated motion field is generated for the whole picture based onunilateral ME. Then the motion field may be used later as CU level orsub-CU level MV candidates.

In some embodiments, the motion field of each reference pictures in bothreference lists is traversed at 4×4 block level. FIG. 20 shows anexample of unilateral Motion Estimation (ME) 2000 in the FRUC method.For each 4×4 block, if the motion associated to the block passingthrough a 4×4 block in the current picture and the block has not beenassigned any interpolated motion, the motion of the reference block isscaled to the current picture according to the temporal distance TD0 andTD1 (the same way as that of MV scaling of TMVP in HEVC) and the scaledmotion is assigned to the block in the current frame. If no scaled MV isassigned to a 4×4 block, the block's motion is marked as unavailable inthe interpolated motion field.

Interpolation and matching cost. When a motion vector points to afractional sample position, motion compensated interpolation is needed.To reduce complexity, bi-linear interpolation instead of regular 8-tapHEVC interpolation can be used for both bilateral matching and templatematching.

The calculation of matching cost is a bit different at different steps.When selecting the candidate from the candidate set at the CU level, thematching cost can be the absolute sum difference (SAD) of bilateralmatching or template matching. After the starting MV is determined, thematching cost C of bilateral matching at sub-CU level search iscalculated as follows:

C=SAD+w·(|MV_(x)−MV_(x) ^(s)|+|MV_(y)−MV_(y) ^(s)|)   Eq. (4)

Here, w is a weighting factor. In some embodiments, w can be empiricallyset to 4. MV and MV^(s) indicate the current MV and the starting MV,respectively. SAD may still be used as the matching cost of templatematching at sub-CU level search.

In FRUC mode, MV is derived by using luma samples only. The derivedmotion will be used for both luma and chroma for MC inter prediction.After MV is decided, final MC is performed using 8-taps interpolationfilter for luma and 4-taps interpolation filter for chroma.

MV refinement is a pattern based MV search with the criterion ofbilateral matching cost or template matching cost. In the JEM, twosearch patterns are supported—an unrestricted center-biased diamondsearch (UCBDS) and an adaptive cross search for MV refinement at the CUlevel and sub-CU level, respectively. For both CU and sub-CU level MVrefinement, the MV is directly searched at quarter luma sample MVaccuracy, and this is followed by one-eighth luma sample MV refinement.The search range of MV refinement for the CU and sub-CU step are setequal to 8 luma samples.

In the bilateral matching merge mode, bi-prediction is applied becausethe motion information of a CU is derived based on the closest matchbetween two blocks along the motion trajectory of the current CU in twodifferent reference pictures. In the template matching merge mode, theencoder can choose among uni-prediction from list0, uni-prediction fromlist1, or bi-prediction for a CU. The selection ca be based on atemplate matching cost as follows:

If costBi <= factor * min (cost0, cost1) bi-prediction is used;Otherwise, if cost0 <= cost1 uni-prediction from list0 is used;Otherwise, uni-prediction from list1 is used;

Here, cost0 is the SAD of list0 template matching, cost1 is the SAD oflist1 template matching and costB1 is the SAD of bi-prediction templatematching. For example, when the value of factor is equal to 1.25, itmeans that the selection process is biased toward bi-prediction. Theinter prediction direction selection can be applied to the CU-leveltemplate matching process.

2.8 Examples of Bi-Directional Optical Flow (BIO)

In BIO, motion compensation is first performed to generate the firstpredictions (in each prediction direction) of the current block. Thefirst predictions are used to derive the spatial gradient, the temporalgradient and the optical flow of each sub-block/pixel within the block,which are then used to generate the second prediction, e.g., the finalprediction of the sub-block/pixel. The details are described as follows.

The bi-directional optical flow (BIO) method is a sample-wise motionrefinement performed on top of block-wise motion compensation forbi-prediction. In some implementations, the sample-level motionrefinement does not use signaling.

Let I^((k)) be the luma value from reference k (k=0, 1) after blockmotion compensation, and denote ∂I^((k))/∂x and ∂I^((k))/∂y as thehorizontal and vertical components of the I^((k)) gradient,respectively. Assuming the optical flow is valid, the motion vectorfield (v_(x), v_(y)) is given by:

∂I ^((k)) /∂t+v _(x) ∂I ^((k)) /∂x+v _(y) ∂I ^((k)) /∂y=0.   Eq. (5)

Combining this optical flow equation with Hermite interpolation for themotion trajectory of each sample results in a unique third-orderpolynomial that matches both the function values I^((k)) and derivatives∂I^((k))/∂x and ∂I^((k))/∂y at the ends. The value of this polynomial att=0 is the BIO prediction:

pred_(BIO)=1/2·(I ⁽⁰⁾ +I ⁽¹⁾ +v _(x)/2·(τ₁ ∂I ⁽¹⁾ /∂x−τ ₀ ∂I ⁽⁰⁾ /∂x)+v_(y)/2·(τ₁ ∂I ⁽¹⁾ /∂y−τ ₀ ∂I ⁽⁰⁾ /∂y)).   Eq. (6)

FIG. 21 shows an example optical flow trajectory in the Bi-directionalOptical flow (BIO) method. Here, τ₀ and τ₁ denote the distances to thereference frames. Distances τ₀ and τ₁ are calculated based on POC forRef₀ and Ref₁: τ₀=POC(current)−POC(Ref₀), τ₁=POC(Ref₁)−POC(current). Ifboth predictions come from the same time direction (either both from thepast or both from the future) then the signs are different (e.g.,τ₀·τ₁<0). In this case, BIO is applied if the prediction is not from thesame time moment (e.g., τ₀≠τ₁). Both referenced regions have non-zeromotion (e.g., MVx₀, MVy₀, MVx₁, MVy₁≠0) and the block motion vectors areproportional to the time distance (e.g., MVx₀/MVx₁=MVy₀/MVy₁=−τ₀/τ₁).

The motion vector field (v_(x), v_(y)) is determined by minimizing thedifference Δ between values in points A and B. FIGS. 9A-9B show anexample of intersection of motion trajectory and reference frame planes.Model uses only first linear term of a local Taylor expansion for Δ:

Δ=(I ⁽⁰⁾ −I ⁽¹⁾ ₀ +v _(x)(τ₁ ∂I ⁽¹⁾ /∂x+τ ₀ ∂I ⁽⁰⁾ /∂x)+v _(y)(τ₁ ∂I ⁽¹⁾/∂y+τ ₀ ∂I ⁽⁰⁾ /∂y))   Eq. (7)

All values in the above equation depend on the sample location, denotedas (i′, j′). Assuming the motion is consistent in the local surroundingarea, Δ can be minimized inside the (2M+1)×(2M+1) square window Ωcentered on the currently predicted point (i, j), where M is equal to 2:

$\begin{matrix}{\left( {v_{x},v_{y}} \right) = {\underset{v_{x},v_{y}}{argmin}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\Delta^{2}\left\lbrack {i^{\prime},j^{\prime}} \right\rbrack}}}} & {{Eq}.\mspace{14mu} (8)}\end{matrix}$

For this optimization problem, the JEM uses a simplified approach makingfirst a minimization in the vertical direction and then in thehorizontal direction. This results in the following:

$\begin{matrix}{\mspace{76mu} {v_{x} = {\left( {s_{1} + r} \right) > {{m?\mspace{14mu} {clip}}\; 3\left( {{- {thBIO}},{thBIO},{- \frac{s_{3}}{\left( {s_{1} + r} \right)}}} \right)\text{:}0}}}} & {{Eq}.\mspace{14mu} (9)} \\{{v_{y} = {\left( {s_{5} + r} \right) > {{m?\mspace{14mu} {clip}}\; 3\left( {{- {thBIO}},{thBIO},{- \frac{s_{6} - {v_{x}s_{2}\text{/}2}}{\left( {s_{5} + r} \right)}}} \right)\text{:}0}}}\mspace{76mu} {{where},}} & {{Eq}.\mspace{14mu} (10)} \\{\mspace{76mu} {{{s_{1} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right)^{2}}};}\mspace{76mu} {{s_{3} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right)}}};}{{s_{2} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}};}\mspace{76mu} {{s_{5} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}};}\mspace{76mu} {s_{6} = {\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}}}} & {{Eq}.\mspace{14mu} (11)}\end{matrix}$

In order to avoid division by zero or a very small value, regularizationparameters r and m can be introduced in Eq. (9) and Eq. (10), where:

r=500·4^(d−8)   Eq. (12)

m=700·4^(d−8)   Eq. (13)

Here, d is bit depth of the video samples.

In order to keep the memory access for BIO the same as for regularbi-predictive motion compensation, all prediction and gradients values,I^((k)), ∂I^((k))/∂x, ∂I^((k))/∂y, are calculated for positions insidethe current block. FIG. 22A shows an example of access positions outsideof a block 2200. As shown in FIG. 22A, in Eq. (9), (2M+1)×(2M+1) squarewindow Ω centered in currently predicted point on a boundary ofpredicted block needs to accesses positions outside of the block. In theJEM, values of I^((k)), ∂I^((k))/∂x , ∂I^((k))/∂y outside of the blockare set to be equal to the nearest available value inside the block. Forexample, this can be implemented as a padding area 2201, as shown inFIG. 22B.

With BIO, it is possible that the motion field can be refined for eachsample. To reduce the computational complexity, a block-based design ofBIO is used in the JEM. The motion refinement can be calculated based ona 4×4 block. In the block-based BIO, the values of s_(n) in Eq. (9) ofall samples in a 4×4 block can be aggregated, and then the aggregatedvalues of s_(n) in are used to derived BIO motion vectors offset for the4×4 block. More specifically, the following formula can used forblock-based BIO derivation:

$\begin{matrix}{\mspace{76mu} {{{s_{1,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in {\Omega {({x,y})}}}\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right)^{2}}}};}{{s_{3,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right)}}}};}{{s_{2,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial x}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial x}}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}}};}\mspace{76mu} {{s_{5,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)^{2}}}};}\mspace{76mu} {s_{6,b_{k}} = {\sum\limits_{{({x,y})} \in b_{k}}{\sum\limits_{{\lbrack{i^{\prime},j}\rbrack} \in \Omega}{\left( {I^{(1)} - I^{(0)}} \right)\left( {{\tau_{1}{\partial I^{(1)}}\text{/}{\partial y}} + {\tau_{0}{\partial I^{(0)}}\text{/}{\partial y}}} \right)}}}}}} & {{Eq}.\mspace{14mu} (14)}\end{matrix}$

Here, b_(k) denotes the set of samples belonging to the k-th 4×4 blockof the predicted block. s_(n) in Eq (9) and Eq (10) are replaced by((s_(n,bk))>>4) to derive the associated motion vector offsets.

In some scenarios, MV regiment of BIO may be unreliable due to noise orirregular motion. Therefore, in BIO, the magnitude of MV regiment isclipped to a threshold value. The threshold value is determined based onwhether the reference pictures of the current picture are all from onedirection. For example, if all the reference pictures of the currentpicture are from one direction, the value of the threshold is set to12×2^(14−d); otherwise, it is set to 12×2^(13−d).

Gradients for BIO can be calculated at the same time with motioncompensation interpolation using operations consistent with HEVC motioncompensation process (e.g., 2D separable Finite Impulse Response (FIR)).In some embodiments, the input for the 2D separable FIR is the samereference frame sample as for motion compensation process and fractionalposition (fracX, fracY) according to the fractional part of block motionvector. For horizontal gradient ∂I/∂x, a signal is first interpolatedvertically using BIOfilterS corresponding to the fractional positionfracY with de-scaling shift d−8. Gradient filter BIOfilterG is thenapplied in horizontal direction corresponding to the fractional positionfracX with de-scaling shift by 18−d. For vertical gradient ∂I/∂y, agradient filter is applied vertically using BIOfilterG corresponding tothe fractional position fracY with de-scaling shift d−8. The signaldisplacement is then performed using BIOfilterS in horizontal directioncorresponding to the fractional position fracX with de-scaling shift by18−d. The length of interpolation filter for gradients calculationBIOfilterG and signal displacement BIOfilterF can be shorter (e.g.,6-tap) in order to maintain reasonable complexity. Table 1 shows examplefilters that can be used for gradients calculation of differentfractional positions of block motion vector in BIO. Table 2 showsexample interpolation filters that can be used for prediction signalgeneration in BIO.

TABLE 1 Exemplary filters for gradient calculations in BIO Fractionalpel position Interpolation filter for gradient(BIOfilterG) 0 {8, −39,−3, 46, −17, 5} 1/16 {8, −32, −13, 50, −18, 5} ⅛  {7, −27, −20, 54, −19,5} 3/16 {6, −21, −29, 57, −18, 5} ¼  {4, −17, −36, 60, −15, 4} 5/16 {3,−9, −44, 61, −15, 4} ⅜  {1, −4, −48, 61, −13, 3} 7/16 {0, 1, −54, 60,−9, 2} ½  {−1, 4, −57, 57, −4, 1}

TABLE 2 Exemplary interpolation filters for prediction signal generationin BIO Interpolation filter for Fractional pel position predictionsignal(BIOfilterS) 0 {0, 0, 64, 0, 0, 0} 1/16 {1, −3, 64, 4, −2, 0} ⅛ {1, −6, 62, 9, −3, 1} 3/16 {2, −8, 60, 14, −5, 1} ¼  {2, −9, 57, 19, −7,2} 5/16 {3, −10, 53, 24, −8, 2} ⅜  {3, −11, 50, 29, −9, 2} 7/16 {3, −11,44, 35, −10, 3} ½  {3, −10, 35, 44, −11, 3}

In the JEM, BIO can be applied to all bi-predicted blocks when the twopredictions are from different reference pictures. When LocalIllumination Compensation (LIC) is enabled for a CU, BIO can bedisabled.

In some embodiments, OBMC is applied for a block after normal MCprocess. To reduce the computational complexity, BIO may not be appliedduring the OBMC process. This means that BIO is applied in the MCprocess for a block when using its own MV and is not applied in the MCprocess when the MV of a neighboring block is used during the OBMCprocess.

2.9 Examples of Decoder-Side Motion Vector Refinement (DMVR)

In a bi-prediction operation, for the prediction of one block region,two prediction blocks, formed using a motion vector (MV) of list0 and aMV of list1, respectively, are combined to form a single predictionsignal. In the decoder-side motion vector refinement (DMVR) method, thetwo motion vectors of the bi-prediction are further refined by abilateral template matching process. The bilateral template matchingapplied in the decoder to perform a distortion-based search between abilateral template and the reconstruction samples in the referencepictures in order to obtain a refined MV without transmission ofadditional motion information.

In DMVR, a bilateral template is generated as the weighted combination(i.e. average) of the two prediction blocks, from the initial MV0 oflist0 and MV1 of list1, respectively, as shown in FIG. 23. The templatematching operation consists of calculating cost measures between thegenerated template and the sample region (around the initial predictionblock) in the reference picture. For each of the two reference pictures,the MV that yields the minimum template cost is considered as theupdated MV of that list to replace the original one. In the JEM, nine MVcandidates are searched for each list. The nine MV candidates includethe original MV and 8 surrounding MVs with one luma sample offset to theoriginal MV in either the horizontal or vertical direction, or both.Finally, the two new MVs, i.e., MV0′ and MV1′ as shown in FIG. 23, areused for generating the final bi-prediction results. A sum of absolutedifferences (SAD) is used as the cost measure.

DMVR is applied for the merge mode of bi-prediction with one MV from areference picture in the past and another from a reference picture inthe future, without the transmission of additional syntax elements. Inthe JEM, when LIC, affine motion, FRUC, or sub-CU merge candidate isenabled for a CU, DMVR is not applied.

3. Examples of CABAC Modifications

In the JEM, CABAC contains the following three major changes compared tothe design in HEVC:

-   -   Modified context modeling for transform coefficients    -   Multi-hypothesis probability estimation with context-dependent        updating speed    -   Adaptive initialization for context models

3.1 Examples of Context Modeling for Transform Coefficients

In HEVC, transform coefficients of a coding block are coded usingnon-overlapped coefficient groups (CGs), and each CG contains thecoefficients of a 4×4 block of a coding block. The CGs inside a codingblock, and the transform coefficients within a CG, are coded accordingto pre-defined scan orders. The coding of transform coefficient levelsof a CG with at least one non-zero transform coefficient may beseparated into multiple scan passes. In the first pass, the first bin(denoted by bin0, also referred as significant_coeff_flag, whichindicates the magnitude of the coefficient is larger than 0) is coded.Next, two scan passes for context coding the second/third bins (denotedby bin1 and bin2, respectively, also referred as coeff_abs_greater1_flagand coeff_abs_greater2_flag) may be applied. Finally, two more scanpasses for coding the sign information and the remaining values (alsoreferred as coeff_abs_level_remaining) of coefficient levels areinvoked, if necessary. Only bins in the first three scan passes arecoded in a regular mode and those bins are termed regular bins in thefollowing descriptions.

In the JEM, the context modeling for regular bins is changed. Whencoding bin i in the i-th scan pass (i being 0, 1, 2), the context indexis dependent on the values of the i-th bins of previously codedcoefficients in the neighbourhood covered by a local template.Specifically, the context index is determined based on the sum of thei-th bins of neighbouring coefficients.

As shown in FIG. 24, the local template contains up to five spatialneighbouring transform coefficients wherein x indicates the position ofcurrent transform coefficient and xi (i being 0 to 4) indicates its fiveneighbours. To capture the characteristics of transform coefficients atdifferent frequencies, one coding block may be split into up to threeregions and the splitting method is fixed regardless of the coding blocksizes. For example, when coding bin0 of luma transform coefficients, asdepicted in FIG. 24, one coding block is split into three regions markedwith different colours, and the context index assigned to each region islisted. Luma and chroma components are treated in a similar way but withseparate sets of context models. Moreover, the context model selectionfor bin0 (e.g., significant flags) of the luma component is furtherdependent on transform size.

3.2 Examples of Multi-Hypothesis Probability Estimation

The binary arithmetic coder is applied with a “multi-hypothesis”probability update model based on two probability estimates P₀ and P₁that are associated with each context model and are updatedindependently with different adaptation rates as follows:

$\begin{matrix}{P_{0}^{new} = \left\{ {{\begin{matrix}{{P_{0}^{old} + \left( {\left( {2^{k} - P_{0}^{old}} \right)\mspace{14mu} \text{>>}\mspace{14mu} M_{i}} \right)},} & {{{if}\mspace{14mu} {input}\mspace{14mu} {is}\mspace{14mu} {‘1’}},} \\{{P_{0}^{old} - \left( {P_{0}^{old}\mspace{14mu} \text{>>}\mspace{14mu} M_{i}} \right)},} & {{{if}\mspace{14mu} {input}\mspace{14mu} {is}\mspace{14mu} {‘0’}},}\end{matrix}P_{1}^{new}} = \left\{ \begin{matrix}{{P_{1}^{old} + \left( {\left( {2^{k} - P_{1}^{old}} \right)\mspace{14mu} \text{>>}\mspace{14mu} 8} \right)},} & {{{if}\mspace{14mu} {input}\mspace{14mu} {is}\mspace{14mu} {‘1’}},} \\{{P_{j}^{old} - \left( {P_{j}^{old}\mspace{14mu} \text{>>}\mspace{14mu} 8} \right)},} & {{if}\mspace{14mu} {input}\mspace{14mu} {is}\mspace{14mu} {{‘0’}.}}\end{matrix} \right.} \right.} & {{Eq}.\mspace{14mu} (15)}\end{matrix}$

Here, P_(j) ^(old) and P_(j) ^(new) (j=0, 1) represent the probabilitiesbefore and after decoding a bin, respectively. The variable M_(i) (being4, 5, 6, 7) is a parameter which controls the probability updating speedfor the context model with index equal to i; and k represents theprecision of probabilities (here it is equal to 15).

The probability estimate P used for the interval subdivision in thebinary arithmetic coder is the average of the estimates from the twohypotheses:

P=(P ₀ ^(new) +P ₁ ^(new))/2   Eq. (16)

In the JEM, the value of the parameter M_(i) used in Equation (15) thatcontrols the probability updating speed for each context model isassigned as follows:

-   -   At the encoder side, the coded bins associated with each context        model are recorded. After one slice is coded, for each context        model with index equal to i, the rate costs of using different        values of M_(i) (being 4, 5, 6, 7) are calculated and the one        that provides the minimum rate cost is selected. For simplicity,        this selection process is performed only when a new combination        of slice type and slice-level quantization parameter are        encountered.    -   A 1-bit flag is signaled for each context model i to indicate        whether M_(i) is different from the default value 4. When the        flag is 1, two bits are used to indicate whether M_(i) is equal        to 5, 6, or 7.

3.3 Examples of Initialization for Context Models

Instead of using fixed tables for context model initialization in HEVC,the initial probability states of context models for inter-coded slicescan be initialized by copying states from previously coded pictures.More specifically, after coding a centrally-located CTU of each picture,the probability states of all context models are stored for potentialuse as the initial states of the corresponding context models on laterpictures. In the JEM, the set of initial states for each inter-codedslice is copied from the stored states of a previously coded picturethat has the same slice type and the same slice-level QP as the currentslice. This lacks loss robustness, but is used in the current JEM schemefor coding efficiency experiment purposes.

4. Examples of Related Embodiments and Methods

Methods related to the disclosed technology include extended LAMVR,wherein supported motion vector resolutions range from ¼-pel to 4-pel(¼-pel, ½-pel, 1-pel, 2-pel and 4-pel). Information about the motionvector resolution is signaled at the CU level when MVD information issignaled.

Depending on the resolution of CU, both motion vector (MV) and motionvector predictor (MVP) of the CU are adjusted. If applied motion vectorresolution is denoted as R (R could be ¼, ½, 1, 2, 4), MV (MV_(x),MV_(y)) and MVP (MVP_(x), MVP_(y)) are represented as follows:

(MV_(x), MV_(y))=(Round(MV_(x)/(R*4))*(R*4), Round(MV_(y)/(R*4))*(R*4))

(MVP_(x), MVP_(y))=(Round(MVP_(x)/(R*4))*(R*4),Round(MVP_(y)/(R*4))*(R*4))

Since the motion vector predictor and MV are both adjusted by anadaptive resolution, MVD (MVD_(x), MVD_(y)) is also aligned to theresolution and, is signaled according to the resolution as follows:

(MVD_(x), MVD_(y))=((MV_(x)−MVP_(x))/(R*4), (MV_(y)−MVP_(y))/R*4))

In this proposal, motion vector resolution index (MVR index) indicatesMVP index as well as the motion vector resolution. As result, theproposed method has no MVP index signaling. The table shows what eachvalue of MVR index represents.

TABLE 3 Examples of MVR index representation MVR Index 0 1 2 3 4Resolution (R) ¼ ½ 1 2 4 in pel N^(th) MVP 1^(st) MVP 2^(nd) MVP 3^(rd)MVP 4^(th) MVP 5^(th) MVP

In the case of bi-prediction, AMVR has 3 modes for every resolution.AMVR Bi-Index indicates whether MVD_(x), MVD_(y) of each reference list(list 0 or list 1) are signaled or not. An example of the AMVR Bi-Indexis defined as in the table below.

TABLE 4 Examples of the AMVR bi-index AMVR Bi-Index (MVD_(x), MVD_(y))of list 0 (MVD_(x), MVD_(y)) of list 1 0 Signaled Signaled 1 Notsignaled Signaled 2 Signaled Not signaled

5. Drawbacks of Existing Implementations

In one existing implementation that uses BIO, the calculated MV betweenreference block/subblock in list 0 (denoted by refblk0) and referenceblock/subblock list 1 (refblk1), denoted by (v_(x), v_(y)), is only usedfor motion compensation of the current block/subblock, and are not usedfor motion prediction, deblocking, OBMC etc. of future coded blocks,which may be inefficient. For example, (v_(x), v_(y)) may be generatedfor each sub-block/pixel of the block, and Eq. (7) may be used togenerate the second prediction of the sub-block/pixel. However, (v_(x),v_(y)) is not used for motion compensation of the sub-block/pixel, whichmay also be inefficient.

In another existing implementation that uses both DMVR and BIO for abi-predicted PU, firstly, DMVR is performed. After that, motioninformation of the PU is updated. Then, BIO is performed with theupdated motion information. That is to say, the input of BIO depends onthe output of DMVR.

In yet another existing implementation that uses OBMC, for AMVP mode,whether OBMC is enabled or not is decided at encoder and signaled todecoder for small blocks (width*height<=256). This increases the encodercomplexity. Meanwhile, for a given block/subblock, when OBMC is enabled,it is always applied to both luma and chroma, this may leads to codingefficiency drop.

In yet another existing implementation that uses AF_INTER mode, MVD isrequired to be coded, however, it can only be encoded in ¼ pelprecision, which may be inefficient.

6. Example Methods for Two-Step Inter-Prediction for Visual Media Coding

Embodiments of the presently disclosed technology overcome the drawbacksof existing implementations, and provide additional solutions, therebyproviding video coding with higher coding efficiencies. The two-stepinter-prediction, based on the disclosed technology, may enhance bothexisting and future video coding standards, is elucidated in thefollowing examples described for various implementations. The examplesof the disclosed technology provided below explain general concepts, andare not meant to be interpreted as limiting. In an example, unlessexplicitly indicated to the contrary, the various features described inthese examples may be combined.

With regard to terminology, the reference picture of current picturefrom list 0 and list 1 is denoted Ref0 and Ref1, respectively. Denoteτ₀=POC(current)−POC(Ref0), τ₁=POC(Ref1)−POC(current), and denotereference block of the current block from Ref0 and Ref1 by refblk0 andrefblk1 respectively. For a sub-block in current block, original MV ofits corresponding sub-block in refblk0 pointing to refblk1 is denoted by(v_(x), v_(y)). MVs of the sub-block in Ref0 and Ref1 are denoted by(mvL0 _(x), mvL0 _(y)) and (mvL1 _(x), mvL1 _(y)) respectively. DerivedMVs derived from the original MVs in BIO are denoted by (v_(x), v_(y)).The updated motion vector based methods for motion prediction may beextended, as described in this patent document, to existing and futurevideo coding standards.

Example 1. MV (v_(x), v_(y)) and MV (mvL1 _(x), mvL1 _(y)), with X=0 or1, should be scaled to the same precision before the adding operation,such as before performing the techniques in Example 1(e) and/or Example2.

-   -   (a) In one example, the target precision (to be scaled to) is        set to be the higher (for better performance)/lower precision        (for lower complexity) between MV (v_(x), v_(y)) and MV        (mvLX_(x), mvLX_(y)). Alternatively, the target precision (to be        scaled to) is set to be a fixed value (e.g., 1/32 pel precision)        regardless the precisions of these two MVs.    -   (b) In one example, the original MV (mvLX_(x), mvLX_(y)) may be        scaled to a higher precision before the adding operation, for        example, it may be scaled from ¼ pel precision to 1/16 pel        precision. In this case,        mvLX_(x)=sign(mvLX_(x))*(abs(mvLX_(x))<<N),        mvLX_(y)=sign(mvLX_(y))*(abs(mvLX_(y))<<N), wherein function        sign( )returns the sign of an input parameter (shown below), and        function abs(·) returns absolute value of an input parameter,        and N=log 2(curr_mv_precision/targ_mv_precision), and        curr_mv_precision and targ_mv_precision are the current MV        precision and target MV precision respectively. For example, if        MV is scaled from ¼ pel precision to 1/16 pel precision, then        N=log 2((¼)/( 1/16))=2.

${{sign}(x)} = \left\{ \begin{matrix}{{- 1},{x < 0}} \\{\mspace{14mu} {1,{x \geq 0}}}\end{matrix} \right.$

-   -   -   (i) Alternatively, mvLX_(x)=mvLX_(x)<<N,            mvLX_(y)=mvLX_(y)<<N.        -   (ii) Alternatively, mvLX_(x)=mvLX_(x)<<(N+K),            mvLX_(y)=mvLX_(y)<<(N+K).        -   (iii) Alternatively,            mvLX_(x)=sign(mvLX_(x))*(abs(mvLX_(x))<<(N+K)),            mvLX_(y)=sign(mvLX_(y))*(abs(mvLX_(y))<<(N+K)).        -   (iv) Similarly, if MV (v_(x), v_(y)) needs to be scaled to a            lower precision, the scaling process as specified in Example            1(d) may be applicable.

    -   (c) In one example, if the precision of MV (v_(x), v_(y)) is        lower/higher than the precision of MV (mvLX_(x), mvLX_(y)), MV        (v_(x), v_(y)) should be scaled to finer/coarser precision. For        example, (mvLX_(x), mvLX_(y)) is with 1/16 pel precision, then        (v_(x), v_(y)) is also scaled to 1/16 pel precision.

    -   (d) If (v_(x), v_(y)) needs to be right-shifted (i.e., scaled to        a lower precision) by N to achieve the same precision with        (mvLX_(x), mvLX_(y)), then v_(x)=(v_(x)+offset)>>N,        v_(y)=(v_(y)+offset)>>N, wherein, for example, offset=1<<(N−1).        -   (i) Alternatively,            v_(x)=sign(v_(x))*((abs(v_(x))+offset)>>N),            v_(y)=sign(v_(y))*((abs(v_(y))+offset)>>N).        -   (ii) Similarly, if MV mvLX_(x), mvLX_(y)) needs to be scaled            to a higher precision, the above scaling process as            specified in Example 1(b) may be applicable.

    -   (e) In one example, it is proposed that the MV (v_(x), v_(y))        derived in BIO is scaled and added up to the original MV        (mvLX_(x), mvLX_(y)) (X=0 or 1) of current block/subblock. The        updated MV is calculated as: mvL0′_(x)=−v_(x)*(τ₀/(τ₀+τ₁))+mvL0        _(x), mvL0′_(y)=−v_(y)*(τ₀/(τ₀+τ₁))+mvL0 _(y), and        mvL1′_(x)=v_(x)*(τ₁/(τ₀+τ₁))+mvL1 _(x),        mvL1′_(y)=v_(y)*(τ₁/(τ₀+τ₁))+mvL1 _(y).        -   (i) In one example, the updated MV is used for future motion            prediction (like in AMVP, merge and affine mode),            deblocking, OBMC etc.        -   (ii) Alternatively, the updated MV can only be used in            motion prediction of its non-immediately following CU/PUs in            decoding order.        -   (iii) Alternatively, the updated MV can only be used as TMVP            in AMVP, merge or affine mode.

    -   (f) If (v_(x), v_(y)) needs to be right-shifted (i.e., scaled to        a lower precision) by N to achieve the same precision with        (mvLX_(x), mvLX_(y)), then v_(x)=(v_(x)+offset)>>(N+K),        v_(y)=(v_(y)+offset)>>(N+K), wherein, for example,        offset=1<<(N+K−1). K is an integer number, for example, K is        equal to 1, 2, 3, −2, −1 or 0.        -   (i) Alternatively,            v_(x)=sign(v_(x))*((abs(v_(x))+offset)>>(N+K)),            v_(y)=sign(v_(y))*((abs(v_(y))+offset)>>(N+K)), wherein, for            example, offset=1<<(N+K−1).

Example 2. Instead of considering POC distances (e.g., in thecomputation of to and ti as described above), the scaling method of MVsinvoked in BIO process may be simplified.

-   -   (a) mvL0′_(x)=−v_(x)/S₀+mvL0 _(x), mvL0′_(y)=−v_(y)/S₀+mvL0        _(y), and/or mvL1′_(x)=v_(x)/S₁+mvL1 _(x),        mvL1′_(y)=v_(y)/S₁+mvL1 _(y). In one example, S₀ and/or S₁ is        set to 2. In one example, it is invoked under certain        conditions, such as τ₀>0 and τ₁>0.        -   (i) Alternatively, offsets may be added during division            process. For example, mvL0′_(x)=(−v_(x)+offset0)/S₀+mvL0            _(x), mvL0′_(y)=−(v_(y)+offset0)/S₀+mvL0 _(y), and/or            mvL1′_(x)=(v_(x)+offset1)/S₁+mvL1 _(x),            mvL1′_(y)=(v_(y)+offset1)/S₁+mvL1 _(y), in one example,            offset0 is set to S₀/2 and offset1 is set to S₁/2.        -   (ii) In one example, mvL0′_(x)=((−v_(x)+1)>>1)+mvL0 _(x),            mvL0′_(y)=(−(v_(y)+1)>>1)+mvL0 _(y), and/or            mvL1′_(x)=((v_(x)+1)>>1)+mvL1 _(x),            mvL1′_(y)=((v_(y)+1)>>1)+mvL1 _(y).    -   (b) mvL0′_(x)=−SF₀*v_(x)+mvL0 _(x), mvL0′_(y)=−v_(y)*SF₀+mvL0        _(y), and/or mvL1′_(x)=−SF₁*v_(x)+mvL1 _(x),        mvL1′_(y)=−SF₁*v_(y)+mvL1 _(y). In one example, SF₀ is set to 2,        and/or SF₁ is set to 1. In one example, it is invoked under        certain conditions, such as τ₀>0 and τ₁<0 and τ₀>|τ₁| as shown        in FIG. 25(b).    -   (c) mvL0′_(x)=SFACT₀*v_(x)+mvL0 _(x),        mvL0′_(y)=SFACT₀*v_(y)+mvL0 _(y), and/or        mvL1′_(x)=SFACT₁*v_(x)+mvL1 _(x), mvL1′_(y)=SFACT₁*v_(y)+mvL1        _(y). In one example, SFACT₀ is set to 1, and/or SFACT₁ is set        to 2. In one example, it is invoked under certain conditions,        such as τ₀>0 and τ₁<0 and τ₀<|τ₁| as shown in FIG. 25(c).

Example 3. Derivation of (v_(x), v_(y)) and update of (mvLX_(x),mvLX_(y)) may be done together when τ₀>0 and τ₁>0 to keep highprecision.

-   -   (a) In one example, if (v_(x), v_(y)) needs to be right-shifted        (i.e., scaled to a lower precision) by N to achieve the same        precision with (mvLX_(x), mvLX_(y)), then        mvL0′_(x)=((−v_(x)+offset)>>(N+1))+mvL0 _(x),        mvL0′_(y)=((−v_(y)+offset)>>(N+1))+mvL0 _(y),        mvL1′_(x)=((v_(x)+offset)>>(N+1))+mvL1 _(x),        mvL1′_(y)=((v_(y)+offset)>>(N+1))+mvL1 _(y), wherein, for        example, offset=1<<N.    -   (b) In one example, if (v_(x), v_(y)) needs to be right-shifted        (i.e., scaled to a lower precision) by N to achieve the same        precision with (mvLX_(x), mvLX_(y)),        mvL0′_(x)=((−v_(x)+offset)>>(N+K+1))+mvL0 _(x),        mvL0′_(y)=((−v_(y)+offset)>>(N+K+1))+mvL0 _(y),        mvL1′_(x)=((v_(x)+offset)>>(N+K+1))+mvL1 _(x),        mvL1′_(y)=((v_(y)+offset)>>(N+K+1))+mvL1 _(y), wherein, for        example, offset=1<<(N+K). K is an integer number, for example, K        is equal to 1, 2, 3, −2, −1 or 0.    -   (c) Alternatively,        mvL0′_(x)=−sign(v_(x))*((abs(v_(x))+offset)>>(N+1))+mvL0 _(x),        mvL0′_(y)=−sign(v_(y))*((abs(v_(y))+offset)>>(N+1))+mvL0 _(y),        mvL1′_(x)=sign(v_(x))*((abs(v_(x))+offset)>>(N+1))+mvL1 _(x),        mvL1′_(y)=sign(v_(y))*((abs(v_(y))+offset)>>(N+1))+mvL1 _(y).    -   (d) Alternatively,        mvL0′_(x)=−sign(v_(x))*((abs(v_(x))+offset)>>(N+K+1))+mvL0 _(x),        mvL0′_(y)=−sign(v_(y))*((abs(v_(y))+offset)>>(N+K+1))+mvL0 _(y),        mvL1′_(x)=sign(v_(x))*((abs(v_(x))+offset)>>(N+K+1))+mvL1 _(x),        mvL1′_(y)=sign(v_(y))*((abs(v_(y))+offset)>>(N+K+1))+mvL1 _(y),        wherein, for example, offset=1<<(N+K). K is an integer number,        for example, K is equal to 1, 2, 3, −2, −1 or 0.

Example 4. Clipping operations may be further applied to the updated MVsemployed in BIO and/or DMVR or other kinds of coding methods that mayrequire MVs to be updated.

-   -   (a) In one example, updated MVs are clipped in the same way        other conventional MVs, e.g., clipped to be within a certain        range compared to picture boundaries.    -   (b) Alternatively, updated MVs are clipped to be within a        certain range (or multiple ranges for different sub-blocks)        compared to the MVs used in MC process. That is, the differences        between the MV used in MC and updated MVs are clipped to be        within a certain range (or multiple ranges for different        sub-blocks).

Example 5. Usage of the updated MV invoked in BIO and/or other kinds ofcoding methods that may require MVs to be updated may be constrained.

-   -   (a) In one example, the updated MV is used for future motion        prediction (like in AMVP, merge and/or affine mode), deblocking,        OBMC etc. Alternatively, updated MVs may be used for a first        module but original MVs may be used for a second module. For        example, the first module is motion prediction, and the second        module is deblocking.        -   (i) In one example, future motion prediction refers to            motion prediction in blocks to be coded/decoded after the            current block in the current picture or slice.        -   (ii) Alternatively, future motion prediction refers to            motion prediction in pictures or slices to be coded/decoded            after the current picture or slice.    -   (b) Alternatively, the updated MV can only be used in motion        prediction of its non-immediately following CU/PUs in decoding        order.    -   (c) The updated MV shall not be used in motion prediction of its        next CU/PUs in decoding order.    -   (d) Alternatively, the updated MV can only be used as predictors        for coding subsequent pictures/tiles/slices, such as TMVP in        AMVP, and/or merge and/or affine mode.    -   (e) Alternatively, the updated MV can only be used as predictors        for coding subsequent pictures/tiles/slices, such as ATMVP        and/or STMVP, etc. al.

Example 6. In one example, a two-step inter-prediction process isproposed wherein the first step is performed to generate someintermediate predictions (first predictions) based on thesignaled/derived motion information associated with the current blockand the second step is performed to derive final predictions of thecurrent block (second predictions) based on updated motion informationwhich may rely on the intermediate predictions.

-   -   (a) In one example, the BIO procedure (i.e., using        signaled/derived motion information which is used to generate        the first predictions and the spatial gradient, the temporal        gradient and the optical flow of each sub-block/pixel within the        block) is only used to derive the updated MV as specified in        Example 1 (and Eq. (7) is not applied to generate the second        prediction), and the updated MV is then used to perform motion        compensation and generate the second prediction (i.e., the final        prediction) of each sub-block/pixel within the block.    -   (b) In one example, different interpolation filters from those        used for inter-coded blocks which are not coded with such a        method may be used in the first or/and the second steps to        reduce the memory bandwidth.        -   (i) In one example, shorter-tap filters (like 6-tap filters,            4-tap filters or bilinear filters) may be used.        -   (ii) Alternatively, the filters (such as filter taps, filter            coefficients) utilized in the first/second steps may be            pre-defined.        -   (iii) Alternatively, furthermore, the selected filter tap            for the first and/or second step may depend on coded            information, such as block sizes/block shapes (square,            non-square, etc. al)/slice types/prediction directions (uni            or bi-prediction or multiple-hypothesis, forward or            backward).        -   (iv) Alternatively, furthermore, different block may choose            different filters for the first/second steps. In one            example, one or more candidate sets of multiple filters may            be pre-defined or signaled. A block may select from the            candidate sets. The selected filter may be indicated by a            signaled index or may be derived on-the-fly without being            signaled.    -   (c) In one example, only integer MV are used when generating the        first predictions and interpolation filter process is not        applied in the first step.        -   (i) In one example, a fractional MV is rounded to the            closest integer MV.            -   (1) If there is more than one closest integer MV, the                fractional MV is rounded to the smaller closest integer                MV.            -   (2) If there is more than one closest integer MV, the                fractional MV is rounded to the larger closest integer                MV.            -   (3) If there is more than one closest integer MV, the                fractional MV is rounded to the closest MV closer to                zero.        -   (ii) In one example, the fractional MV is rounded to the            closest integer MV that is no smaller than the fractional            MV.        -   (iii) In one example, the fractional MV is rounded to the            closest integer MV that is no larger than the fractional MV.    -   (d) Usage of such a method may be signaled in SPS, PPS, Slice        header, CTUs or CUs or group of CTUs.    -   (e) Usage of such a method may further depend on coded        information, such as block sizes/block shapes (square,        non-square, etc. al)/slice types/prediction directions (uni or        bi-prediction or multiple-hypothesis, forward or backward).        -   (i) In one example, such a method may be automatically            disallowed under certain conditions, for example, it may be            disabled when the current block is coded with the affine            mode.        -   (ii) In one example, such a method may be automatically            applied under certain conditions, such as when the block is            coded with bi-prediction and block size is larger than a            threshold (e.g., more than 16 samples).

Example 7. In one example, it is proposed that before calculating thetemporal gradient in BIO, a reference block (or a prediction block) maybe modified firstly, and the calculation of temporal gradient is basedon the modified reference block.

-   -   (a) In one example, mean is removed for all reference blocks.        -   (i) For example, for reference block X with X=0 or 1, first,            mean (denoted by MeanX) is calculated for the block, then            each pixel in the reference block is subtracted by MeanX.        -   (ii) Alternatively, for different reference picture list, it            may decide whether to remove the mean or not. For example,            for one reference block/sub-block, mean is removed before            calculating temporal gradients, while for another one, mean            is not removed.        -   (iii) Alternatively, different reference blocks (e.g., 3 or            4 reference blocks utilized in multiple hypothesis            prediction) may select whether to be modified firstly or            not.    -   (b) In one example, mean is defined as the average of selected        samples in the reference block.    -   (c) In one example, all pixels in a reference block X or a        sub-block of the reference block X are used to calculate MeanX.    -   (d) In one example, only partial pixels in a reference block X        or a sub-block of the reference block are used to calculate        MeanX. For example, only pixels in every second row/column are        used.        -   (i) Alternatively, and in an example, only pixels in every            fourth row/column are used to calculate MeanX.        -   (ii) Alternatively, only four corner pixels are used to            calculate MeanX.        -   (iii) Alternatively, only four corner pixels and the center            pixel, for example, pixel at position (W/2, H/2) wherein W×H            is the reference block size, are used to calculate MeanX.    -   (e) In one example, reference blocks may be firstly filtered        before being used to derive temporal gradients.        -   (i) In one example, smooth filtering methods may be first            applied to reference blocks.        -   (ii) In one example, the pixels at block boundaries are            first filtered.        -   (iii) In one example, Overlapped Block Motion Compensation            (OBMC) is first applied before deriving temporal gradients.        -   (iv) In one example, Illumination Compensation (IC) is first            applied before deriving temporal gradients.        -   (v) In one example, weighted prediction is first applied            before deriving temporal gradients.    -   (f) In one example, temporal gradient is calculated first and        then is modified. For example, temporal gradient is further        subtracted by the difference between Mean0 and Mean1.

Example 8. In one example, whether to update MV for BIO coded blocksand/or use updated MV for future motion prediction and/or how to useupdated MV for future motion prediction may be signaled from the encoderto the decoder, such as in Video Parameter Set (VPS), Sequence ParameterSet (SPS), Picture Parameter Set (PPS), Slice header, CTUs or CUs.

Example 9. In one example, it is proposed to add a constraint to motionvectors utilized in the BIO process.

-   -   (a) In one example, (v_(x), v_(y)) are constrained to given        ranges, −M_(x)<v_(x)<N_(x), and/or −M_(y)<v_(y)<N_(y), wherein        M_(x), N_(x), M_(y), N_(y) are none-negative integers, and may        be equal to 32, for example.    -   (b) In one example, updated MVs of a BIO-coded sub-block/a        BIO-coded block are constrained to given ranges, such as        −M_(L0x)<mvL0′_(x)<N_(L0x) and/or −M_(L1x)<mvL1′_(x)<N_(L1x),        −M_(L0y)<mvL0′_(y)<N_(L0y) and/or −M_(L1y)<mvL1′_(y)<N_(L1y),        wherein M_(L0x), N_(L0x), M_(L1x), N_(L1x), M_(L0y), N_(L0y),        M_(L1y), N_(L1y) are none-negative integers, and may be equal to        1024, 2048 and so on, for example.

Example 10. It is proposed that for BIO, DMVR, FRUC, template matchingor other methods that require to update MV (or motion informationincluding MV and/or reference pictures) from those derived from thebitstream, usage of the updated motion information may be constrained.

-   -   (a) In one example, even if motion information is updated at        block level, updated and non-updated motion information may be        stored differently for different subblocks. In one example, the        updated motion information of some subblocks may be stored and        for the other remaining subblocks, the non-updated motion        information are stored.    -   (b) In one example, if MV (or motion information) is updated at        subblock/block level, the updated MV are only stored for inner        subblocks, i.e., subblocks that are not at PU/CU/CTU boundary,        and are then used for motion prediction, deblocking, OBMC etc.        as shown in FIGS. 26A and 26B. Alternatively, the updated MV are        only stored for boundary subblocks.    -   (c) In one example, the updated motion information from a        neighbouring block is not used if the neighbouring block and the        current block are not in the same CTU or the same region with a        size such as 64×64 or 32×32.        -   (i) In one example, a neighbouring block is treated as            “unavailable” if the neighbouring block and the current            block are not in the same CTU or the same region with a size            such as 64×64 or 32×32.        -   (ii) Alternatively, the motion information without the            updating process is used by the current block if the            neighbouring block and the current block are not in the same            CTU or the same region with a size such as 64×64 or 32×32.    -   (d) In one example, the updated MVs from a neighbouring block        are not used if the neighbouring block and the current block are        not in the same CTU row or the same row of regions with a size        such as 64×64 or 32×32.        -   (i) In one example, a neighbouring block is treated as            “unavailable” if the neighbouring block and the current            block are not in the same CTU row or the same row of regions            with a size such as 64×64 or 32×32.        -   (ii) Alternatively, the motion information without the            updating process is used by the current block if the            neighbouring block and the current block are not in the same            CTU row or the same row of regions with a size such as 64×64            or 32×32.    -   (e) In one example, the motion information of a block is not        updated if the bottom-most row of the block is the bottom-most        row of a CTU or a region of a size such as 64×64 or 32×32.    -   (f) In one example, the motion information of a block is not        updated if the right-most column of the block is the right-most        column of a CTU or a region of a size such as 64×64 or 32×32.    -   (g) In one example, refined motion information from some        neighboring CTU or region are used for the current CTU, and        unrefined motion information from other neighboring CTU or        regions are used for the current CTU.        -   (i) In one example, refined motion information from left CTU            or left region are used for current CTU.        -   (ii) Alternatively, in addition, refined motion information            from top-left CTU or top-left region are used for current            CTU.        -   (iii) Alternatively, in addition, refined motion information            from top CTU or top region are used for current CTU.        -   (iv) Alternatively, in addition, refined motion information            from top-right CTU or top-right region are used for current            CTU.        -   (v) In one example, region is of size such as 64×64 or            32×32.

Example 11. In one example, it is proposed that different MVD precisionsmaybe used in AF_INTER mode and a syntax element may be signaled toindicate the MVD precision for each block/CU/PU.

-   -   (a) In one example, {¼, 1, 4} pel MVD precision are allowed.    -   (b) In one example, {¼, ½, 1, 2, 4} pel MVD precision are        allowed.    -   (c) In one example, { 1/16, ⅛, ¼} pel MVD precision are allowed.    -   (d) The syntax element is present under further conditions, such        as when there is non-zero MVD component of the block/CU/PU.    -   (e) In one example, the MVD precision information is always        signaled regardless whether there is any non-zero MVD component        or not.    -   (f) Alternatively, for 4/6-parameter AF_INTER mode wherein ⅔        MVDs are encoded, different MVD precisions may be used for the ⅔        MVDs, (1 MVD for each control point in uni-prediction and 2 MVDs        for each control point in bi-prediction, i.e., 1 MVD for each        control point in each prediction direction), and the ⅔ control        points are associated with different MVD precisions. In this        case, furthermore, ⅔ syntax elements may be signaled to indicate        the MVD precisions.    -   (g) In one example, the methods described in PCT/CN2018/091792        may be used for coding the MVD precisions in AF_INTER mode.

Example 12. In one example, it is proposed that different Decoder-sideMotion Vector Derivation (DMVD) methods like BIO, DMVR, FRUC andtemplate matching etc. work independently if more than one DMVD methodsare performed for a block (e.g., PU), i.e., the input of a DMVD methoddoes not depend on the output of another DMVD method.

-   -   (a) In one example, furthermore, one prediction block and/or one        set of updated motion information (e.g., motion vector and        reference picture for each prediction direction) are generated        from the multiple sets of motion information derived by the        multiple DMVD methods.    -   (b) In one example, motion compensation is performed using the        derived motion information of each DMVD method, and they are        averaged or weighted averaged or filtered (like by median        filter) to generate the final prediction.    -   (c) In one example, the derived motion information of all DMVD        methods are averaged or weighted averaged or filtered (like by        median filter) to generate the final motion information.        Alternatively, different priorities are assigned to different        DMVD methods, and motion information derived by the method with        the highest priority is selected as the final motion        information. For example, when both BIO and DMVR are performed        for a PU, then motion information generated by DMVR is used as        the final motion information.    -   (d) In one example, no more than N DMVD method are allowed for a        PU, wherein N>=1.        -   (i) Different priorities are assigned to different DMVD            methods, and the methods that are valid and are with the            highest N priority are performed.    -   (e) The DMVD methods are performed in a simultaneous way. The        updated MV of one DMVD method is not input as the start-point of        the next DMVD method. For all DMVD methods, the non-updated MV        is input as the searching start-point. Alternatively, the DMVD        methods are performed in a cascade way. The updated MV of one        DMVD method is input as the searching start-point of the next        DMVD method.

Additional Embodiments

This section describes the way of MV refinement and stored for futureusage of BIO coded blocks. The refined MVs may be used for motion vectorprediction of following blocks within current slice/CTU row/tile, and/orfiltering process (e.g., deblocking filter process) and/or motion vectorprediction for blocks located at different pictures.

As depicted in FIG. 32, the derived motion vector, pointing from thesub-block in reference block 0 to the sub-block in reference block 1,denoted by (DMV_(x), DMV_(y)), is used to further improve prediction ofthe current sub-block.

It is proposed to further refine the motion vector of each sub-block byusing the derived motion vector in BIO. Denote the POC distance (e.g.,absolute POC difference) between the LX reference picture and thecurrent picture as deltaPOCX, and denote (MVLX_(x), MVLX_(y)) and(MVLX_(x)′, MVLX_(y)′) as the signaled and refined motion vector of thecurrent sub-block, with X=0 or 1. Then (MVLX_(x)′, MVLX_(y)′) iscalculated as follows:

${{MVL}\; 0_{x}^{\prime}} = {{{MVL}\; 0_{x}} - \frac{{deltaPOC}\; 0*{DMV}_{x}}{\left( {{{deltaPOC}\; 0} + {{deltaPOC}\; 1}} \right)}}$${{MVL}\; 0_{y}^{\prime}} = {{{MVL}\; 0_{y}} - \frac{{deltaPOC}\; 0*{DMV}_{y}}{\left( {{deltaPOCX} + {{deltaPOC}\; 1}} \right)}}$${{MVL}\; 1_{x}^{\prime}} = {{{MVL}\; 1_{x}} + \frac{{deltaPOC}\; 1*{DMV}_{x}}{\left( {{{deltaPOC}\; 0} + {{deltaPOC}\; 1}} \right)}}$${{MVL}\; 1_{y}^{\prime}} = {{{MVL}\; 1_{y}} + \frac{{deltaPOC}\; 1*{DMV}_{y}}{\left( {{deltaPOCX} + {{deltaPOC}\; 1}} \right)}}$

However, multiplication and division are required in the aboveequations. To address this issue, derivation of the refined motionvector is simplified as follows:

MVL0′_(x)=MVL0_(x)−((DMV_(x)+1)>>1)

MVL0′_(y)=MVL0_(y)−((DMV_(y)+1)>>1)

MVL1′_(x)=MVL1_(x)+((DMV_(x)+1)>>1)

MVL1′_(y)=MVL1_(y)+((DMV_(y)+1)>>1)

In some embodiments, this method is employed only when the current CU ispredicted from a preceding picture and a following picture, andtherefore only works in the Random Access (RA) configuration.

Example 13. The proposed method may be applied under certain conditions,such as block sizes, slice/picture/tile types.

-   -   (a) In one example, when a block size contains smaller than M*H        samples, e.g., 16 or 32 or 64 luma samples, the above methods        are not allowed.    -   (b) Alternatively, when minimum size of a block's width or        height is smaller than or no larger than X, the above methods        are not allowed. In one example, X is set to 8.    -   (c) Alternatively, when a block's width>th1 or >=th1 and/or a        block's height>th2 or >=th2, the above methods are not allowed.        In one example, X is set to 8.    -   (d) Alternatively, when a block's width<th1 or <=th1 and/or a        block's height<th2 or <a=th2, the above methods are not allowed.        In one example, X is set to 8.

Example 14. The above methods may be applied in a sub-block level.

-   -   (a) In one example, the BIO updating process, or a two-step        inter prediction process or a temporal gradient derivation        method described in Example 7, may be invoked for each        sub-block.    -   (b) In one example, when a block with either width or height or        both width and height are both larger than (or equal to) a        threshold L, the block may be split into multiple sub-blocks.        Each sub-block is treated in the same way as a normal coding        block with size equal to the sub-block size.

Example 15. The threshold may be pre-defined or signaled inSPS/PPS/picture/slice/tile level.

-   -   (a) Alternatively, the thresholds may be depend on certain coded        information, such as block size, picture type, temporal layer        index, etc.

The examples described above may be incorporated in the context of themethods described below, e.g., methods 2700-3100, 3300-3600 and3800-4200, which may be implemented at a video decoder.

FIG. 27 shows a flowchart of an exemplary method for video decoding. Themethod 2700 includes, at step 2710, receiving a bitstream representationof a current block of video data.

The method 2700 includes, at step 2720, generating updated first andsecond reference motion vectors based on a weighted sum of a firstscaled motion vector and first and second scaled reference motionvectors, respectively. In some embodiments, the first scaled motionvector is generated by scaling a first motion vector to a targetprecision, and wherein the first and second scaled reference motionvectors are generated by scaling first and second reference motionvectors to the target precision, respectively. In some embodiments, thefirst motion vector is derived based on the first reference motionvector from a first reference block and the second reference motionvector from a second reference block, and where the current block isassociated with the first and second reference blocks.

In some embodiments, an indication of the target precision is signaledin a Video Parameter Set (VPS), a Sequence Parameter Set (SPS), aPicture Parameter Set (PPS), a slice header, a coding tree unit (CTU) ora coding unit (CU).

In some embodiments, the first motion vector has a first precision andthe first and second reference motion vectors have a referenceprecision. In other embodiments, the first precision may be higher orlower than the reference precision. In yet other embodiments, the targetprecision may be set to either the first precision, the referenceprecision or a fixed (or predetermined) precision regardless of thefirst and reference precisions.

In some embodiments, the first motion vector is derived based on abi-directional optical flow (BIO) refinement using the first and secondreference motion vectors.

The method 2700 includes, at step 2730, processing the bitstreamrepresentation based on the updated first and second reference motionvectors to generate the current block. In some embodiments, theprocessing is based on a bi-directional optical flow (BIO) refinement ora decoder-side motion vector refinement (DMVR), and wherein the updatedfirst and second reference motion vectors are clipped prior to theprocessing.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement, and the updated first and second reference motionvectors are constrained to a predetermined range of values prior to theprocessing.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement, a decoder-side motion vector refinement (DMVR),frame-rate up conversion (FRUC) techniques or a template matchingtechnique. In one example, the updated first and second reference motionvectors are generated for inner sub-blocks that are not on a boundary ofthe current block. In another example, the updated first and secondreference motion vectors are generated for a subset of sub-blocks of thecurrent block.

In some embodiments, the processing is based on at least two techniques,which may include a bi-directional optical flow (BIO) refinement, adecoder-side motion vector refinement (DMVR), frame-rate up conversion(FRUC) techniques or a template matching technique. In one example, theprocessing is performed for each of the at least two techniques togenerate multiple sets of results, which may be averaged or filtered togenerate the current block. In another example, the processing isperformed in a cascaded manner for each of the at least two techniquesto generate the current block.

FIG. 28 shows a flowchart of an exemplary method for video decoding. Themethod 2800 includes, at step 2810, generating, for a current block, anintermediate prediction based on a first motion information associatedwith the current block. In some embodiments, generating the intermediateprediction comprises a first interpolation filtering process. In someembodiments, the generating the intermediate prediction is further basedon signaling in a sequence parameter set (SPS), a picture parameter set(PPS), a coding tree unit (CTU), a slice header, a coding unit (CU) or agroup of CTUs.

The method 2800 includes, at step 2820, updating the first motioninformation to a second motion information. In some embodiments,updating the first motion information comprises using a bi-directionaloptical flow (BIO) refinement.

The method 2800 includes, at step 2830, generating a final predictionfor the current block based on the intermediate prediction or the secondmotion information. In some embodiments, generating the final predictioncomprises a second interpolation filtering process.

In some embodiments, the first interpolation filtering process uses afirst set of filters that are different from a second set of filtersused by the second interpolation filtering process. In some embodiments,at least one filter tap of the first or second interpolation filteringprocess is based on a dimension, a prediction direction, or a predictiontype of the current block.

FIG. 29 shows a flowchart of another exemplary method for videodecoding. This example includes some features and/or steps that aresimilar to those shown in FIG. 28, and described above. At least some ofthese features and/or components may not be separately described in thissection.

The method 2900 includes, at step 2910, receiving a bitstreamrepresentation of a current block of video data. In some embodiments,step 2910 includes receiving the bitstream representation from a memorylocation or buffer in a video encoder or decoder. In other embodiments,step 2910 includes receiving the bitstream representation over awireless or wired channel at a video decoder. In yet other embodiments,step 2910 include receiving the bitstream representation from adifferent module, unit or processor, which may implement one or moremethods as described in, but not limited to, the embodiments in thepresent document.

The method 2900 includes, at step 2920, generating intermediate motioninformation based on motion information associated with the currentblock.

The method 2900 includes, at step 2930, generating updated first andsecond reference motion vectors based on first and second referencemotion vectors, respectively. In some embodiments, the current block isassociated with first and second reference blocks. In some embodiments,the first and second reference motion vectors are associated with thefirst and second reference blocks, respectively.

The method 2900 includes, at step 2940, processing the bitstreamrepresentation based on the intermediate motion information or theupdated first and second reference motion vectors to generate thecurrent block.

In some embodiments of method 2900, the generating the updated first andsecond reference motion vectors is based on a weighted sum of a firstscaled motion vector and first and second scaled reference motionvectors, respectively. In some embodiments, a first motion vector isderived based on the first reference motion vector and the secondreference motion vector, the first scaled motion vector is generated byscaling the first motion vector to a target precision, and the first andsecond scaled reference motion vectors are generated by scaling thefirst and second reference motion vectors to the target precision,respectively.

In some embodiments, an indication of the target precision is signaledin a Video Parameter Set (VPS), a Sequence Parameter Set (SPS), aPicture Parameter Set (PPS), a slice header, a coding tree unit (CTU) ora coding unit (CU).

In some embodiments, the first motion vector has a first precision andthe first and second reference motion vectors have a referenceprecision. In other embodiments, the first precision may be higher orlower than the reference precision. In yet other embodiments, the targetprecision may be set to either the first precision, the referenceprecision or a fixed (or predetermined) precision regardless of thefirst and reference precisions.

In some embodiments, the first motion vector is derived based on abi-directional optical flow (BIO) refinement using the first and secondreference motion vectors.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement, and the updated first and second reference motionvectors are constrained to a predetermined range of values prior to theprocessing.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement or a decoder-side motion vector refinement (DMVR),and wherein the updated first and second reference motion vectors areclipped prior to the processing.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement, a decoder-side motion vector refinement (DMVR),frame-rate up conversion (FRUC) techniques or a template matchingtechnique. In one example, the updated first and second reference motionvectors are generated for inner sub-blocks that are not on a boundary ofthe current block. In another example, the updated first and secondreference motion vectors are generated for a subset of sub-blocks of thecurrent block.

In some embodiments, the processing is based on at least two techniques,which may include a bi-directional optical flow (BIO) refinement, adecoder-side motion vector refinement (DMVR), frame-rate up conversion(FRUC) techniques or a template matching technique. In one example, theprocessing is performed for each of the at least two techniques togenerate multiple sets of results, which may be averaged or filtered togenerate the current block. In another example, the processing isperformed in a cascaded manner for each of the at least two techniquesto generate the current block.

FIG. 30 shows a flowchart of an exemplary method for video decoding. Themethod 3000 includes, at step 3010, generating, for a current block, anintermediate prediction based on a first motion information associatedwith the current block. In some embodiments, generating the intermediateprediction comprises a first interpolation filtering process. In someembodiments, the generating the intermediate prediction is further basedon signaling in a sequence parameter set (SPS), a picture parameter set(PPS), a coding tree unit (CTU), a slice header, a coding unit (CU) or agroup of CTUs.

The method 3000 includes, at step 3020, updating the first motioninformation to a second motion information. In some embodiments,updating the first motion information comprises using a bi-directionaloptical flow (BIO) refinement.

The method 3000 includes, at step 3030, generating a final predictionfor the current block based on the intermediate prediction or the secondmotion information. In some embodiments, generating the final predictioncomprises a second interpolation filtering process.

In some embodiments, the first interpolation filtering process uses afirst set of filters that are different from a second set of filtersused by the second interpolation filtering process. In some embodiments,at least one filter tap of the first or second interpolation filteringprocess is based on a dimension, a prediction direction, or a predictiontype of the current block.

FIG. 31 shows a flowchart of another exemplary method for videodecoding. This example includes some features and/or steps that aresimilar to those shown in FIG. 30, and described above. At least some ofthese features and/or components may not be separately described in thissection.

The method 3100 includes, at step 3110, receiving a bitstreamrepresentation of a current block of video data. In some embodiments,step 3110 includes receiving the bitstream representation from a memorylocation or buffer in a video encoder or decoder. In other embodiments,step 3110 includes receiving the bitstream representation over awireless or wired channel at a video decoder. In yet other embodiments,step 3110 include receiving the bitstream representation from adifferent module, unit or processor, which may implement one or moremethods as described in, but not limited to, the embodiments in thepresent document.

The method 3100 includes, at step 3120, generating intermediate motioninformation based on motion information associated with the currentblock.

The method 3100 includes, at step 3130, generating updated first andsecond reference motion vectors based on first and second referencemotion vectors, respectively. In some embodiments, the current block isassociated with first and second reference blocks. In some embodiments,the first and second reference motion vectors are associated with thefirst and second reference blocks, respectively.

The method 3100 includes, at step 3140, processing the bitstreamrepresentation based on the intermediate motion information or theupdated first and second reference motion vectors to generate thecurrent block.

In some embodiments of method 3100, the generating the updated first andsecond reference motion vectors is based on a weighted sum of a firstscaled motion vector and first and second scaled reference motionvectors, respectively. In some embodiments, a first motion vector isderived based on the first reference motion vector and the secondreference motion vector, the first scaled motion vector is generated byscaling the first motion vector to a target precision, and the first andsecond scaled reference motion vectors are generated by scaling thefirst and second reference motion vectors to the target precision,respectively.

In some embodiments, an indication of the target precision is signaledin a Video Parameter Set (VPS), a Sequence Parameter Set (SPS), aPicture Parameter Set (PPS), a slice header, a coding tree unit (CTU) ora coding unit (CU).

In some embodiments, the first motion vector has a first precision andthe first and second reference motion vectors have a referenceprecision. In other embodiments, the first precision may be higher orlower than the reference precision. In yet other embodiments, the targetprecision may be set to either the first precision, the referenceprecision or a fixed (or predetermined) precision regardless of thefirst and reference precisions.

In some embodiments, the first motion vector is derived based on abi-directional optical flow (BIO) refinement using the first and secondreference motion vectors.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement, and the updated first and second reference motionvectors are constrained to a predetermined range of values prior to theprocessing.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement or a decoder-side motion vector refinement (DMVR),and wherein the updated first and second reference motion vectors areclipped prior to the processing.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement, a decoder-side motion vector refinement (DMVR),frame-rate up conversion (FRUC) techniques or a template matchingtechnique. In one example, the updated first and second reference motionvectors are generated for inner sub-blocks that are not on a boundary ofthe current block. In another example, the updated first and secondreference motion vectors are generated for a subset of sub-blocks of thecurrent block.

In some embodiments, the processing is based on at least two techniques,which may include a bi-directional optical flow (BIO) refinement, adecoder-side motion vector refinement (DMVR), frame-rate up conversion(FRUC) techniques or a template matching technique. In one example, theprocessing is performed for each of the at least two techniques togenerate multiple sets of results, which may be averaged or filtered togenerate the current block. In another example, the processing isperformed in a cascaded manner for each of the at least two techniquesto generate the current block.

FIG. 33 shows a flowchart of an exemplary method for video decoding. Themethod 3300 includes, at step 3310, generating, for a current block, anintermediate prediction based on a first motion information associatedwith the current block. In some embodiments, generating the intermediateprediction comprises a first interpolation filtering process. In someembodiments, the generating the intermediate prediction is further basedon signaling in a sequence parameter set (SPS), a picture parameter set(PPS), a coding tree unit (CTU), a slice header, a coding unit (CU) or agroup of CTUs.

The method 3300 includes, at step 3320, updating the first motioninformation to a second motion information. In some embodiments,updating the first motion information comprises using a bi-directionaloptical flow (BIO) refinement.

The method 3300 includes, at step 3330, generating a final predictionfor the current block based on the intermediate prediction or the secondmotion information. In some embodiments, generating the final predictioncomprises a second interpolation filtering process.

In some embodiments, the first interpolation filtering process uses afirst set of filters that are different from a second set of filtersused by the second interpolation filtering process. In some embodiments,at least one filter tap of the first or second interpolation filteringprocess is based on a dimension, a prediction direction, or a predictiontype of the current block.

FIG. 34 shows a flowchart of another exemplary method for videodecoding. This example includes some features and/or steps that aresimilar to those shown in FIG. 33, and described above. At least some ofthese features and/or components may not be separately described in thissection.

The method 3400 includes, at step 3410, receiving a bitstreamrepresentation of a current block of video data. In some embodiments,step 3410 includes receiving the bitstream representation from a memorylocation or buffer in a video encoder or decoder. In other embodiments,step 3410 includes receiving the bitstream representation over awireless or wired channel at a video decoder. In yet other embodiments,step 3410 include receiving the bitstream representation from adifferent module, unit or processor, which may implement one or moremethods as described in, but not limited to, the embodiments in thepresent document.

The method 3400 includes, at step 3420, generating intermediate motioninformation based on motion information associated with the currentblock.

The method 3400 includes, at step 3430, generating updated first andsecond reference motion vectors based on first and second referencemotion vectors, respectively. In some embodiments, the current block isassociated with first and second reference blocks. In some embodiments,the first and second reference motion vectors are associated with thefirst and second reference blocks, respectively.

The method 3400 includes, at step 3440, processing the bitstreamrepresentation based on the intermediate motion information or theupdated first and second reference motion vectors to generate thecurrent block.

In some embodiments of method 3400, the generating the updated first andsecond reference motion vectors is based on a weighted sum of a firstscaled motion vector and first and second scaled reference motionvectors, respectively. In some embodiments, a first motion vector isderived based on the first reference motion vector and the secondreference motion vector, the first scaled motion vector is generated byscaling the first motion vector to a target precision, and the first andsecond scaled reference motion vectors are generated by scaling thefirst and second reference motion vectors to the target precision,respectively.

In some embodiments, an indication of the target precision is signaledin a Video Parameter Set (VPS), a Sequence Parameter Set (SPS), aPicture Parameter Set (PPS), a slice header, a coding tree unit (CTU) ora coding unit (CU).

In some embodiments, the first motion vector has a first precision andthe first and second reference motion vectors have a referenceprecision. In other embodiments, the first precision may be higher orlower than the reference precision. In yet other embodiments, the targetprecision may be set to either the first precision, the referenceprecision or a fixed (or predetermined) precision regardless of thefirst and reference precisions.

In some embodiments, the first motion vector is derived based on abi-directional optical flow (BIO) refinement using the first and secondreference motion vectors.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement, and the updated first and second reference motionvectors are constrained to a predetermined range of values prior to theprocessing.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement or a decoder-side motion vector refinement (DMVR),and wherein the updated first and second reference motion vectors areclipped prior to the processing.

In some embodiments, the processing is based on a bi-directional opticalflow (BIO) refinement, a decoder-side motion vector refinement (DMVR),frame-rate up conversion (FRUC) techniques or a template matchingtechnique. In one example, the updated first and second reference motionvectors are generated for inner sub-blocks that are not on a boundary ofthe current block. In another example, the updated first and secondreference motion vectors are generated for a subset of sub-blocks of thecurrent block.

In some embodiments, the processing is based on at least two techniques,which may include a bi-directional optical flow (BIO) refinement, adecoder-side motion vector refinement (DMVR), frame-rate up conversion(FRUC) techniques or a template matching technique. In one example, theprocessing is performed for each of the at least two techniques togenerate multiple sets of results, which may be averaged or filtered togenerate the current block. In another example, the processing isperformed in a cascaded manner for each of the at least two techniquesto generate the current block.

FIG. 35 shows a flowchart of an exemplary method for video decoding. Themethod 3500 includes, at step 3510, generating, for a bitstreamrepresentation of a current block, an updated reference block bymodifying a reference block associated with the current block.

In some embodiments, the method 3500 further includes the step offiltering the reference block using a smoothing filter.

In some embodiments, the method 3500 further includes the step offiltering pixels at block boundaries of the reference block.

In some embodiments, the method 3500 further includes the step ofapplying overlapped block motion compensation (OBMC) to the referenceblock.

In some embodiments, the method 3500 further includes the step ofapplying illumination compensation (IC) to the reference block.

In some embodiments, the method 3500 further includes the step ofapplying a weighted prediction to the reference block.

The method 3500 includes, at step 3520, calculating, based on theupdated reference block, a temporal gradient for a bi-directionaloptical flow (BIO) motion refinement.

The method 3500 includes, at step 3530, performing, based on thetemporal gradient, a conversion, which includes the BIO motionrefinement, between the bitstream representation and the current block.In some embodiments, the conversion generates the current block from thebitstream representation (e.g., as might be implemented in a videodecoder). In other embodiments, the conversion generates the bitstreamrepresentation from the current block (e.g., as might be implemented ina video encoder).

In some embodiments, the method 3500 further includes the steps ofcomputing a mean value for the reference block, and subtracting the meanvalue from each pixel of the reference block. In one example, computingthe mean value is based on all pixels of the reference block. In anotherexample, computing the mean value is based on all pixels in a sub-blockof the reference block.

In some embodiments, computing the mean value is based on a subset ofpixels (in other words, not all the pixels) of the reference block. Inone example, the subset of pixels includes pixels in every fourth row orcolumn of the reference block. In another example, the subset of pixelsincludes four corner pixels. In yet another example, the subset ofpixels includes the four corner pixels and a center pixel.

FIG. 36 shows a flowchart of another exemplary method for videodecoding. This example includes some features and/or steps that aresimilar to those shown in FIG. 35, and described above. At least some ofthese features and/or components may not be separately described in thissection.

The method 3600 includes, at step 3610, generating, for a bitstreamrepresentation of a current block, a temporal gradient for abi-directional optical flow (BIO) motion refinement.

The method 3600 includes, at step 3620, generating an updated temporalgradient by subtracting a difference of a first mean value and a secondmean value from the temporal gradient, where the first mean value is amean value for a first reference block, the second mean value is a meanvalue for a second reference block, and the first and second referenceblocks are associated with the current block.

In some embodiments, the mean value is based on all pixels of thecorresponding reference block (e.g., the first mean value is computed asthe average of all the pixels of the first reference block). In anotherexample, computing the mean value is based on all pixels in a sub-blockof the corresponding reference block.

In some embodiments, the mean value is based on a subset of pixels (inother words, not all the pixels) of the corresponding reference block.In one example, the subset of pixels includes pixels in every fourth rowor column of the corresponding reference block. In another example, thesubset of pixels includes four corner pixels. In yet another example,the subset of pixels includes the four corner pixels and a center pixel.

The method 3600 includes, at step 3630, performing, based on the updatedtemporal gradient, a conversion, which includes the BIO motionrefinement, between the bitstream representation and the current block.In some embodiments, the conversion generates the current block from thebitstream representation (e.g., as might be implemented in a videodecoder). In other embodiments, the conversion generates the bitstreamrepresentation from the current block (e.g., as might be implemented ina video encoder).

FIG. 38 shows a flowchart of an exemplary method for video processing.The method 3800 includes, at step 3810, determining original motioninformation for a current block; at step 3820, scaling original motionvectors of the original motion information and derived motion vectorsderived based on the original motion vectors to a same target precision;at step 3830, generating updated motion vectors from the scaled originaland derived motion vectors; and at step 3840, performing a conversionbetween the current block and a bitstream representation of a videoincluding the current block, based on the updated motion vectors.

FIG. 39 shows a flowchart of an exemplary method for video processing.The method 3900 includes, at step 3910, determining original motioninformation for a current block; at step 3920, updating original motionvectors of the original motion information for the current block basedon a refining method; at step 3930, clipping the updated motion vectorsto be within a range; and at step 3940, performing a conversion betweenthe current block and a bitstream representation of a video includingthe current block, based on the clipped updated motion vectors.

FIG. 40 shows a flowchart of an exemplary method for video processing.The method 4000 includes, at step 4010, determining original motioninformation associated with a current block; at step 4020, generatingupdated motion information based on specific prediction mode; and atstep 4030, performing, based on the updated motion information, aconversion between the current block and a bitstream representation of avideo data including the current block, wherein the specific predictionmode includes one or more of bi-directional optical flow (BIO)refinement, a decoder-side motion vector refinement (DMVR), frame-rateup conversion (FRUC) techniques or a template matching technique.

FIG. 41 shows a flowchart of an exemplary method for video processing.The method 4100 includes, at step 4110, determining a motion vectordifference (MVD) precision for a current block processed with affinemode from an MVD precision set; at step 4120, performing, based on thedetermined MVD precision, a conversion between the current block and abitstream representation of a video including the current block.

FIG. 42 shows a flowchart of an exemplary method for video processing.The method 4200 includes, at step 4210, determining non-updated motioninformation associated with a current block; at step 4220, updating thenon-updated motion information based on multiple decoder-side motionvector derivation (DMVD) methods to generate updated motion informationfor the current block; at step 4230, performing, based on the updatedmotion information, a conversion between the current block and abitstream representation of a video including the current block.

7. Example Implementations of the Disclosed Technology

FIG. 37 is a block diagram of a video processing apparatus 3700. Theapparatus 3700 may be used to implement one or more of the methodsdescribed herein. The apparatus 3700 may be embodied in a smartphone,tablet, computer, Internet of Things (IoT) receiver, and so on. Theapparatus 3700 may include one or more processors 3702, one or morememories 3704 and video processing hardware 3706. The processor(s) 3702may be configured to implement one or more methods (including, but notlimited to, methods 2700-3100, 3300-3600 and 3800-4200) described in thepresent document. The memory (memories) 3704 may be used for storingdata and code used for implementing the methods and techniques describedherein. The video processing hardware 3706 may be used to implement, inhardware circuitry, some techniques described in the present document.

In some embodiments, the video coding methods may be implemented usingan apparatus that is implemented on a hardware platform as describedwith respect to FIG. 37.

Various embodiments and techniques described throughout the presentdocument may be described using the following clause based format.

1.1. A video processing method, comprising: determining original motioninformation for a current block; scaling original motion vectors of theoriginal motion information and derived motion vectors derived based onthe original motion vectors to a same target precision; generatingupdated motion vectors from the scaled original and derived motionvectors; and performing a conversion between the current block and abitstream representation of a video including the current block, basedon the updated motion vectors.

1.2. The method of example 1.1, wherein the original motion vectors havea first precision, the derived motion vectors have a second precisiondifferent from the first precision, and the target precision is set to ahigher precision or a lower precision between the first precision andthe second precision.

1.3. The method of example 1.1, wherein the target precision is set to afixed precision.

1.4. The method of example 1.1, wherein the target precision is higherthan a precision of the original motion vectors.

1.5. The method of example 1.4, wherein the original motion vectors arescaled as:

mvLX′_(x)=sign(mvLX_(x))*(abs(mvLX_(x))<<N),

mvLX′_(y)=sign(mvLX_(y))*(abs(mvLX_(y))<<N),

wherein (mvLX_(x), mvLX_(y)) are the original motion vectors,(mvLX′_(x), mvLX′_(y)) are the scaled original motion vectors, functionsign(⋅) returns a sign of an input parameter, function abs(⋅) returnsabsolute value of an input parameter, N=log2(curr_mv_precision/targ_mv_precision), and wherein curr_mv_precision isthe precision of the original motion vectors, and targ_mv_precision is aprecision of the derived motion vectors as the target precision.

1.6. The method of example 1.1, wherein the target precision is the sameas a precision of the original motion vectors.

1.7. The method of example 1.1, wherein the original motion vectors havea first precision, the derived motion vectors have a second precisiondifferent from the first precision, and the target precision is set tothe first precision.

1.8. The method of example 1.7, wherein, the derived motion vectors arescaled as:

v′_(x)=(v_(x)+offset)>>N, v′_(y)=(v_(y)+offset)>>N; or

v′_(x)=sign(v_(x))*((abs(v_(x))+offset)>>N),v′_(y)=sign(v_(y))*((abs(v_(y))+offset)>>N)

when the derived motion vectors are to be right-shifted by N to achievethe target precision, wherein (v_(x), v_(y)) are the derived motionvectors, (v′_(x), v′_(y)) are the scaled derived motion vectors, offsetis an offset applied to the derived motion vectors to achieve the targetprecision, function sign(⋅) returns a sign of an input parameter,function abs(⋅) returns absolute value of an input parameter, N=log2(curr_mv_precision/targ_mv_precision), wherein curr_mv_precision is thefirst precision, and targ_mv_precision is the second precision.

1.9. The method of example 1.1, wherein the scaling and the generationof the updated motion vectors are performed as:

mvL0′_(x)=−v_(x)/S₀+mvL0 _(x), mvL0′_(y)=−v_(y)/S₀+mvL0 _(y); and/or

mvL1′_(x)=v_(x)/S₁+mvL1 _(x), mvL1′_(y)=v_(y)/S₁+mvL1 _(y)

wherein, (mvL0 _(x), mvL0 _(y)) and (mvL1 _(x), mvL1 _(y)) are theoriginal motion vectors, (mvL0′_(x), mvL0′_(y)) and (mvL1′_(x),mvL1′_(y)) are the updated motion vectors, (v_(x), v_(y)) are thederived motion vectors, and S₀ and S₁ are scaling factors.

1.10. The method of example 1.1, wherein the scaling and the generationof the updated motion vectors are performed as:

mvL0′_(x)=(−v_(x)+offset0)/S₀+mvL0 _(x),mvL0′_(y)=−(v_(y)+offset0)/S₀+mvL0 _(y), and/or

mvL1′_(x)=(v_(x)+offset1)/S₁+mvL1 _(x),mvL1′_(y)=(v_(y)+offset1)/S₁+mvL1 _(y)

wherein, (mvL0 _(x), mvL0 _(y)) and (mvL1 _(x), mvL1 _(y)) are theoriginal motion vectors, (mvL0′_(x), mvL0′_(y)) and (mvL1′_(x),mvL1′_(y)) are the updated motion vectors, (v_(x), v_(y)) are thederived motion vectors, offset0 and offset1 are offsets, and S₀ and S₁are scaling factors.

1.11. The method of example 1.1, wherein the scaling and the generationof the updated motion vectors are performed as:

mvL0′_(x)=((−v_(x)+1)>>1)+mvL0 _(x), mvL0′_(y)=(−(v_(y)+1)>>1)+mvL0_(y), and/or

mvL1′_(x)=((v_(x)+1)>>1)+mvL1 _(x), mvL1′_(y)=((v_(y)+1)>>1)+mvL1 _(y)

wherein, (mvL0 _(x), mvL0 _(y)) and (mvL1 _(x), mvL1 _(y)) are theoriginal motion vectors, (mvL0′_(x), mvL0′_(y)) and (mvL1′_(x),mvL1′_(y)) are the updated motion vectors, and (v_(x), v_(y)) are thederived motion vectors.

1.12. The method of anyone of examples 1.9-1.11, wherein the scaling andthe generation of the updated motion vectors are performed when τ₀>0 andτ₁>0, wherein τ₀=POC(current)−POC(Ref₀), τ₁=POC(Ref₁)−POC(current), andwherein POC(current), POC(Ref₀) and POC(Ref₁) are the picture ordercounts of the current block, a first reference block and a secondreference block, respectively.

1.13. The method of example 1.1, wherein the scaling and the generationof the updated motion vectors are performed as:

mvL0′_(x)=−SF₀*v_(x)+mvL0 _(x), mvL0′_(y)=−v_(y)*SF₀+mvL0 _(y), and/or

mvL1′_(x)=−SF₁*v_(x)+mvL1 _(x), mvL1′_(y)=−SF₁*v_(y)+mvL1 _(y)

wherein, (mvL0 _(x), mvL0 _(y)) and (mvL1 _(x), mvL1 _(y)) are theoriginal motion vectors, (mvL0′_(x), mvL0′_(y)) and (mvL1′_(x),mvL1′_(y)) are the updated motion vectors, (v_(x), v_(y)) are thederived motion vectors, and SF₀ and SF₁ are scaling factors.

1.14. The method of example 1.13, wherein the scaling and the generationof the updated motion vectors are performed when τ₀>0, τ₁<0 and τ₀>|τ₁|,wherein τ₀=POC(current)−POC(Ref₀), τ₁=POC(Ref₁)−POC(current), andwherein POC(current), POC(Ref₀) and POC(Ref₁) are the picture ordercounts of the current block, a first reference block and a secondreference block, respectively.

1.15. The method of example 1.1, wherein the scaling and the generationof the updated motion vectors are performed as:

mvL0′_(x)=SFACT₀*v_(x)+mvL0 _(x), mvL0′_(y)=SFACT₀*v_(y)+mvL0 _(y),and/or

mvL1′_(x)=SFACT₁*v_(x)+mvL1 _(x), mvL1′_(y)=SFACT₁*v_(y)+mvL1 _(y)

wherein, (mvL0 _(x), mvL0 _(y)) and (mvL1 _(x), mvL1 _(y)) are theoriginal motion vectors, (mvL0′_(x), mvL0′_(y)) and (mvL1′_(x),mvL1′_(y)) are the updated motion vectors, (v_(x), v_(y)) are thederived motion vectors, and SFACT₀ and SFACT₁ are scaling factors.

1.16. The method of example 1.15, wherein the scaling and the generationof the updated motion vectors are performed when τ₀>0, τ₁<0 and τ<|τ₁|,wherein τ₀=POC(current)−POC(Ref₀), τ₁=POC(Ref₁)−POC(current), andwherein POC(current), POC(Ref₀) and POC(Ref₁) are the picture ordercounts of the current block, a first reference block and a secondreference block, respectively.

1.17. The method of example 1.1, wherein the derivation of the derivedmotion vectors and the generation of the updated motion vectors areperformed together when τ₀>0 and τ₁>0, whereinτ₀=POC(current)−POC(Ref₀), τ₁=POC(Ref₁)−POC(current), and whereinPOC(current), POC(Ref₀) and POC(Ref₁) are the picture order counts ofthe current block, a first reference block and a second reference block,respectively.

1.18. The method of example 1.17, wherein the scaling and the generationof the updated motion vectors are performed as:

mvL0′_(x)=((−v_(x)+offset)>>(N+1))+mvL0 _(x),mvL0′_(y)=((−v_(y)+offset1)>>(N+1))+mvL0 _(y),mvL1′_(x)=((v_(x)+offset)>>(N+1))+mvL1 _(x),mvL1′_(y)=((v_(y)+offset2)>>(N+1))+mvL1 _(y),

when the derived motion vectors are right-shifted by N to achieve thetarget precision, wherein (mvL0 _(x), mvL0 _(y)) and (mvL1 _(x), mvL1_(y)) are the original motion vectors, (mvL0′_(x), mvL0′_(y)) and(mvL1′_(x), mvL1′_(y)) are the updated motion vectors, and (v_(x),v_(y)) are the derived motion vectors, offset1 and offset 2 are offsets,N=log 2(curr_mv_precision/targ_mv_precision), and whereincurr_mv_precision is a precision of the original motion vectors, andtarg_mv_precision is a precision of the derived motion vectors.

1.19. The method of example 1.17, wherein the original motion vectorshave a first precision, the derived motion vectors have a secondprecision different from the first precision, and the original motionvectors are left-shifted by N to achieve the target precision as thesecond precision.

1.20. The method of example 1.17, wherein the original motion vectorsare left-shifted by K and the derived motion vectors are right-shiftedby N-K to achieve the target precision.

1.21. The method of example 1.17, wherein, the scaling and thegeneration of the updated motion vectors are performed as:

mvL0′_(x)=−sign(v_(x))*((abs(v_(x))+offset0)>>(N+1))+mvL0 _(x),

mvL0′_(y)=−sign(v_(y))*((abs(v_(y))+offset0)>>(N+1))+mvL0 _(y),

mvL1′_(x)=sign(v_(x))*((abs(v_(x))+offset1)>>(N+1))+mvL1 _(x),

mvL1′_(y)=sign(v_(y))*((abs(v_(y))+offset1)>>(N+1))+mvL1 _(y)

wherein (mvL0 _(x), mvL0 _(y)) and (mvL1 _(x), mvL1 _(y)) are theoriginal motion vectors, (mvL0′_(x), mvL0′_(y)) and (mvL1′_(x),mvL1′_(y)) are the updated motion vectors, (v_(x), v_(y)) are thederived motion vectors, offset0 and offset1 are offsets, functionsign(⋅) returns a sign of an input parameter, function abs(⋅) returnsabsolute value of an input parameter, N=log2(curr_mv_precision/targ_mv_precision), curr_mv_precision is a precisionof the original motion vectors, and targ_mv_precision is a precision ofthe derived motion vector.

1.22. The method of example 1.1, wherein the updating of the first andsecond reference motion vectors comprises using a bi-directional opticalflow (BIO) refinement.

1.23. The method of anyone of examples 1.1-1.22, wherein the method isnot applied in case that the current block meets a certain condition.

1.24. The method of example 1.23, wherein the certain conditionspecifies at least one of: a size of the current block, a slice type ofthe current block, a picture type of the current block and a tile typeof the current block.

1.25. The method of example 1.23, wherein the certain conditionspecifies that a sample number that the current block contains issmaller than a first threshold.

1.26. The method of example 1.23, wherein the certain conditionspecifies that minimum size of a width and a height of the current blockis smaller than or no larger than a second threshold.

1.27. The method of example 1.23, wherein the certain conditionspecifies that a width of the current block is smaller than or no largerthan a third threshold, and/or a height of the current block is smallerthan or no larger than a fourth threshold.

1.28. The method of example 1.23, wherein the certain conditionspecifies that a width of the current block is larger than or no smallerthan a third threshold, and/or a height of the current block is largerthan or no smaller than a fourth threshold.

1.29. The method of example 1.23, wherein the method is applied in asub-block level, in case that a width and/or a height of a block, towhich the sub-block belongs, is equal to or larger than a fifththreshold.

1.30. The method of example 1.29, wherein the current block is splitinto multiple sub-blocks and each of the multiple sub-blocks furtherundergoes a bidirectional optical flow (BIO) process, in a same way as anormal coding block with a size equal to the sub-block size.

1.31. The method of anyone of examples 1.25-1.29, wherein each of thefirst to fifth thresholds is pre-defined or signaled in sequenceparameter set (SPS) level, or picture parameter set (PPS) level, orpicture level, or slice level, or tile level.

1.32. The method of example 1.31, wherein each of the first to fifththresholds is defined depending on coded information including at leastone of a block size, a picture type, and a temporal layer index.

1.33. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement the method in any one of examples 1.1 to 1.32.

1.34. A computer program product stored on a non-transitory computerreadable media, the computer program product including program code forcarrying out the method in any one of examples 1.1 to 1.32.

2.1. A video processing method, comprising: determining original motioninformation for a current block; updating original motion vectors of theoriginal motion information for the current block based on a refiningmethod; clipping the updated motion vectors to be within a range; andperforming a conversion between the current block and a bitstreamrepresentation of a video including the current block, based on theclipped updated motion vectors.

2.2. The method of example 2.1, wherein the refining method comprises abi-directional optical flow (BIO) refinement, a decoder-side motionvector refinement (DMVR) , a frame-rate up conversion (FRUC) or atemplate matching.

2.3. The method of example 2.1, wherein the updated motion vectors areclipped to be within a range that is same as an allowed range of theoriginal motion vectors.

2.4. The method of example 2.1, wherein differences between the updatedmotion vectors and the original motion vectors are clipped to be withina same range or different ranges for different sub-blocks.

2.5. The method of example 2.1, wherein the refining method comprises abi-directional optical flow (BIO) refinement, and motion vectors derivedfrom the original motion vectors in the BIO refinement are constrainedto a first range as:

−M_(x)<v_(x)<N_(x), and/or −M_(y)<v_(y)<N_(y),

wherein (v_(x), v_(y)) are the derived motion vectors, and M_(x), N_(x),M_(y), N_(y) are none-negative integers.

2.6. The method of example 2.1, wherein the refining method comprises abi-directional optical flow (BIO) refinement, and the updated motionvectors are constrained to a second range as:

−M_(L0x)<mvL0′_(x)<N_(L0x) and/or

−M_(L1x)<mvL1′_(x)<N_(L1x), and/or

−M_(L0y)<mvL0′_(y)<N_(L0y) and/or

−M_(L1y)<mvL1′_(y)<N_(L1y)

wherein (mvL0′_(x), mvL0′_(y)) and (mvL1′_(x), mvL1′_(y)) are theupdated motion vectors for different reference list, and M_(L0x),N_(L0x), M_(L1x), N_(L1x), M_(L0y), N_(L0y), M_(L1y), N_(L1y) arenone-negative integers.

2.7. The method of anyone of examples 2.1-2.6, wherein the method is notapplied in case that the current block meets a certain condition.

2.8. The method of example 2.7, wherein the certain condition specifiesat least one of: a size of the current block, a slice type of thecurrent block, a picture type of the current block and a tile type ofthe current block.

2.9. The method of example 2.7, wherein the certain condition specifiesthat a sample number that the current block contains is smaller than afirst threshold.

2.10. The method of example 2.7, wherein the certain condition specifiesthat minimum size of a width and a height of the current block issmaller than or no larger than a second threshold.

2.11. The method of example 2.7, wherein the certain condition specifiesthat a width of the current block is smaller than or no larger than athird threshold, and/or a height of the current block is smaller than orno larger than a fourth threshold.

2.12. The method of example 2.7, wherein the certain condition specifiesthat a width of the current block is larger than or no smaller than athird threshold, and/or a height of the current block is larger than orno smaller than a fourth threshold.

2.13. The method of example 2.7, wherein the method is applied in asub-block level, in case that a width and/or a height of a block, towhich the sub-block belongs, is equal to or larger than a fifththreshold.

2.14. The method of example 2.13, wherein the current block is splitinto multiple sub-blocks and each of the multiple sub-blocks furtherundergoes a bidirectional optical flow (BIO) process, in a same way as anormal coding block with a size equal to the sub-block size.

2.15. The method of anyone of examples 2.9-2.13, wherein each of thefirst to fifth thresholds is pre-defined or signaled in sequenceparameter set (SPS) level, or picture parameter set (PPS) level, orpicture level, or slice level, or tile level.

2.16. The method of example 2.15, wherein each of the first to fifththresholds is defined depending on coded information including at leastone of a block size, a picture type, and a temporal layer index.

2.17. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement the method in any one of examples 2.1 to 2.16.

2.18. A computer program product stored on a non-transitory computerreadable media, the computer program product including program code forcarrying out the method in any one of examples 2.1 to 2.16.

3.1. A method for video processing, comprising: determining originalmotion information associated with a current block; generating updatedmotion information based on specific prediction mode; and performing,based on the updated motion information, a conversion between thecurrent block and a bitstream representation of a video data includingthe current block, wherein the specific prediction mode includes one ormore of bi-directional optical flow (BIO) refinement, a decoder-sidemotion vector refinement (DMVR), frame-rate up conversion (FRUC)techniques or a template matching technique.

3.2. The method of example 3.1, wherein the updated motion informationcomprises updated motion vectors.

3.3. The method of example 3.1, wherein the updated motion vectors areused for motion prediction for coding subsequent video blocks; or theupdated motion vectors are used for filtering or overlapped block motioncompensation (OBMC).

3.4. The method of example 3.2, wherein the updated motion vectors areused for motion prediction in advanced motion vector prediction (AMVP)mode, merge mode and/or affine mode.

3.5. The method of example 3.3, wherein the filtering includesdeblocking filtering.

3.6. The method of any one of examples 3.1-3.5, wherein the updatedmotion information are used for a first module and original motioninformation are used for a second module.

3.7. The method of example 3.6, wherein the first module is motionprediction module, and the second module is deblocking module.

3.8. The method of any one of examples 3.2-3.7, wherein the motionprediction is used in processing blocks after the current block in acurrent picture or slice.

3.9. The method of any one of examples 3.2-3.7, wherein the motionprediction is used in processing pictures or slices to be processedafter a current picture or slice including the current block.

3.10. The method of any one of examples 3.1-3.9, wherein the updatedmotion vectors are only used in motion information prediction of codingunits (CUs) or prediction units (Pus) that are not immediately followingthe current block in processing order.

3.11. The method of any one of examples 3.1-3.10, wherein the updatedmotion vectors are not used in motion prediction of CUs/PUs that arenext to the current block in processing order.

3.12. The method of any one of examples 3.1-3.11, wherein the updatedmotion vectors are only used as predictors for processing subsequentpictures/tiles/slices.

3.13. The method of example 3.12, wherein the updated motion vectors areused as temporal motion vector prediction (TMVP) in advanced motionvector prediction (AMVP) mode, merge mode or affine mode.

3.14. The method of example 3.12, wherein the updated motion vectors areonly used as predictors for processing subsequent pictures/tiles/slicesin alternative temporal motion vector prediction (ATMVP) mode and/orspatial-temporal motion vector prediction (STMVP) mode.

3.15. The method of any one of examples 3.1-3.14, wherein informationincluding whether to update MV for BIO coded blocks and/or use theupdated MV for motion prediction and/or how to use the updated MV formotion prediction is signaled from an encoder to a decoder.

3.16. The method of example 3.15, further comprising that theinformation is signaled in Video Parameter Set (VPS), Sequence ParameterSet (SPS), Picture Parameter Set (PPS), Slice header, coding tree units(CTUs) or CUs.

3.17. The method of example 3.1, further comprising updating motioninformation that includes updating motion vector and reference picturefor each prediction direction at block level.

3.18. The method of example 3.1 or 3.17, wherein the updated motioninformation is stored for some sub-blocks and for the other remainingsub-blocks, the non-updated motion information is stored within oneblock.

3.19. The method of example 3.1 or 3.17, the updated motion vectors areonly stored for inner sub-blocks that are not at PU/CU/CTU boundary.

3.20. The method of example 3.19, further comprising the updated motionvectors for inner sub-blocks are used for motion prediction, deblocking,or OBMC.

3.21. The method of example 3.1 or 3.17, the updated motion vectors areonly stored for boundary sub-blocks that are at PU/CU/CTU boundary.

3.22. The method of example 3.1 or 3.17, wherein updated motioninformation from a neighboring block is not used if the neighboringblock and the current block are not in a same CTU or a same region witha size of 64×64 or 32×32.

3.23. The method of example 3.22, wherein the neighboring block ismarked as unavailable if the neighboring block and the current block arenot in the same CTU or the same region with a size of 64×64 or 32×32.

3.24. The method of example 3.22, wherein non-updated motion informationis used by the current block if the neighboring block and the currentblock are not in the same CTU or the same region with a size of 64×64 or32×32.

3.25. The method of example 3.17, wherein the updated motion vectorsfrom a neighboring block are not used if the neighboring block and thecurrent block are not in a same CTU row or a same row of regions with asize of 64×64 or 32×32.

3.26. The method of example 3.25, wherein the a neighboring block ismarked as unavailable if the neighboring block and the current block arenot in the same CTU row or the same row of regions with a size of 64×64or 32×32.

3.27. The method of example 3.25, wherein non-updated motion informationfrom the neighboring block is used by the current block if theneighboring block and the current block are not in the same CTU row orthe same row of regions with a size of 64×64 or 32×32.

3.28. The method of example 3.17, wherein the motion information of ablock is not updated if a bottom-most row of the block is a bottom-mostrow of a CTU or a region with a size of 64×64 or 32×32.

3.29. The method of example 3.17, wherein the motion information of ablock is not updated if a right-most column of the block is a right-mostcolumn of a CTU or a region with a size of 64×64 or 32×32.

3.30. The method of example 3.1 or 3.17, further comprising predictingmotion information of blocks/CUs within the current CTU based on theupdated motion information or the non-updated motion information ofneighboring CTUs or regions.

3.31. The method of example 3.30, wherein the updated motion informationfrom left CTU or left region are used for the current CTU.

3.32. The method of example 3.30 or 3.31, wherein the updated motioninformation from top-left CTU or top-left region are used for thecurrent CTU.

3.33. The method of any one of examples 3.30-3.32, wherein the updatedmotion information from top CTU or top region are used for the currentCTU.

3.34. The method of any one of examples 3.30-3.33, wherein the updatedmotion information from top-right CTU or top-right region are used forthe current CTU.

3.35. The method of any one of examples 3.30-3.34, wherein each of theone or more regions have a size of 64×64 or 32×32.

3.36. The method of anyone of examples 3.1-3.35, wherein the method isnot applied in case that the current block meets a certain condition.

3.37. The method of example 3.36, wherein the certain conditionspecifies at least one of: a size of the current block, a slice type ofthe current block, a picture type of the current block and a tile typeof the current block.

3.38. The method of example 3.36, wherein the certain conditionspecifies that a sample number that the current block contains issmaller than a first threshold.

3.39. The method of example 3.36, wherein the certain conditionspecifies that minimum size of a width and a height of the current blockis smaller than or no larger than a second threshold.

3.40. The method of example 3.36, wherein the certain conditionspecifies that a width of the current block is smaller than or no largerthan a third threshold, and/or a height of the current block is smallerthan or no larger than a fourth threshold.

3.41. The method of example 3.36, wherein the certain conditionspecifies that a width of the current block is larger than or no smallerthan a third threshold, and/or a height of the current block is largerthan or no smaller than a fourth threshold.

3.42. The method of example 3.36, wherein the method is applied in asub-block level, in case that a width and/or a height of a block, towhich the sub-block belongs, is equal to or larger than a fifththreshold.

3.43. The method of example 3.42, wherein the current block is splitinto multiple sub-blocks and each of the multiple sub-blocks furtherundergoes a bidirectional optical flow (BIO) process, in a same way as anormal coding block with a size equal to the sub-block size.

3.44. The method of anyone of examples 3.38-3.42, wherein each of thefirst to fifth thresholds is pre-defined or signaled in sequenceparameter set (SPS) level, or picture parameter set (PPS) level, orpicture level, or slice level, or tile level.

3.45. The method of example 3.44, wherein each of the first to fifththresholds is defined depending on coded information including at leastone of a block size, a picture type, and a temporal layer index.

3.46. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement the method in any one of examples 3.1 to 3.45.

3.47. A computer program product stored on a non-transitory computerreadable media, the computer program product including program code forcarrying out the method in any one of examples 3.1 to 3.45.

4.1. A video processing method, comprising: determining a motion vectordifference (MVD) precision for a current block processed with affinemode from an MVD precision set; performing, based on the determined MVDprecision, a conversion between the current block and a bitstreamrepresentation of a video including the current block.

4.2. The method of example 4.1, wherein the MVD represents a differencebetween a predicted motion vector and an actual motion vector usedduring motion compensation processing.

4.3. The method of example 4.2, wherein the MVD precision set includesmultiple different MVD precisions which construct a geometric sequence.

4.4. The method of example 4.3, wherein the MVD precision set includes¼, 1 and 4 pixel MVD precisions.

4.5. The method of example 4.3, wherein the MVD precision set includes¼, ½, 1, 2 and 4 pixel MVD precisions.

4.6. The method of example 4.3, wherein the MVD precision set includes1/16, ⅛ and ¼ pixel MVD precisions.

4.7. The method of example 4.1, wherein the current block is a codingunit or a prediction unit.

4.8. The method of any one of examples 4.1-4.7, wherein determining theMVD precision further comprises: determining the MVD precision for thecurrent block based on a syntax element indicating the MVD precision.

4.9. The method of example 4.8, wherein the syntax element is presentwhen there is non-zero MVD component of the current block.

4.10. The method of example 4.8, wherein the syntax element is notpresent when there is no non-zero MVD component of the current block.

4.11. The method of example 4.8, wherein the syntax element is presentregardless whether there is any non-zero MVD component of the currentblock.

4.12. The method of example 4.1, wherein the current block is processedwith affine inter mode or affine advanced motion vector prediction(AMVP) mode.

4.13. The method of example 4.12, wherein different MVDs of the currentblock are associated with different MVD precisions.

4.14. The method of example 4.13, wherein the affine inter mode is a4-parameter affine inter mode with 2 control points, and one MVD is usedfor each control point in each prediction direction.

4.15. The method of example 4.14, wherein the 2 control points areassociated with different MVD precisions.

4.16. The method of example 4.13, wherein the affine inter mode is a6-parameter affine inter mode with 3 control points, and one MVD is usedfor each control point in each prediction direction.

4.17. The method of example 4.16, wherein the 3 control points areassociated with different MVD precisions.

4.18. The method of example 4.15, wherein two syntax elements arepresent to indicate the different MVD precisions for the 2 controlpoints.

4.19. The method of example 4.17, wherein three syntax elements arepresent to indicate the different MVD precisions for the 3 controlpoints.

4.20. The method of example 4.1, wherein the MVD precision set isdetermined based on coding information of the current block.

4.21. The method of example 4.20, wherein the coding informationincludes a quantization level of the current block.

4.22. The method of example 4.21, wherein a set of coarser MVD precisionvalues is selected for a larger quantization level.

4.23. The method of example 4.21, wherein a set of finer MVD precisionvalues is selected for a smaller quantization level.

4.24. The method of anyone of examples 4.1-4.23, wherein the method isnot applied in case that the current block meets a certain condition.

4.25. The method of example 4.24, wherein the certain conditionspecifies at least one of: a size of the current block, a slice type ofthe current block, a picture type of the current block and a tile typeof the current block.

4.26. The method of example 4.24, wherein the certain conditionspecifies that a sample number that the current block contains issmaller than a first threshold.

4.27. The method of example 4.24, wherein the certain conditionspecifies that minimum size of a width and a height of the current blockis smaller than or no larger than a second threshold.

4.28. The method of example 4.24, wherein the certain conditionspecifies that a width of the current block is smaller than or no largerthan a third threshold, and/or a height of the current block is smallerthan or no larger than a fourth threshold.

4.29. The method of example 4.24, wherein the certain conditionspecifies that a width of the current block is larger than or no smallerthan a third threshold, and/or a height of the current block is largerthan or no smaller than a fourth threshold.

4.30. The method of example 4.24, wherein the method is applied in asub-block level, in case that a width and/or a height of a block, towhich the sub-block belongs, is equal to or larger than a fifththreshold.

4.31. The method of example 4.30, wherein the current block is splitinto multiple sub-blocks and each of the multiple sub-blocks furtherundergoes a bidirectional optical flow (BIO) process, in a same way as anormal coding block with a size equal to the sub-block size.

4.32. The method of anyone of examples 4.26-4.30, wherein each of thefirst to fifth thresholds is pre-defined or signaled in sequenceparameter set (SPS) level, or picture parameter set (PPS) level, orpicture level, or slice level, or tile level.

4.33. The method of example 4.32, wherein each of the first to fifththresholds is defined depending on coded information including at leastone of a block size, a picture type, and a temporal layer index.

4.34. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement the method in any one of examples 4.1 to 4.33.

4.35. A computer program product stored on a non-transitory computerreadable media, the computer program product including program code forcarrying out the method in any one of examples 4.1 to 4.33.

5.1. A video processing method, comprising: determining non-updatedmotion information associated with a current block; updating thenon-updated motion information based on multiple decoder-side motionvector derivation (DMVD) methods to generate updated motion informationfor the current block; and performing, based on the updated motioninformation, a conversion between the current block and a bitstreamrepresentation of a video including the current block.

5.2. The method of example 5.1, wherein the multiple DMVD methodsinclude at least two of the followings: bi-directional optical flow(BIO) refinement, decoder-side motion vector refinement (DMVR),frame-rate up conversion (FRUC) technique, and template matchingtechnique.

5.3. The method of example 5.2, wherein the multiple DMVD methods areperformed in a simultaneous way on the non-updated motion informationfor the current block, and a non-updated motion vector of thenon-updated motion information is input as the searching start-point foreach of the multiple DMVD methods.

5.4. The method of example 5.2, wherein the multiple DMVD methods areperformed in a cascade way on the non-updated motion information for thecurrent block, and an updated motion vector of the updated motioninformation generated by one DMVD method is input as the searchingstart-point of a next DMVD method.

5.5. The method of example 5.4, wherein the one DMVD method is DMVR andthe next DMVD method is BIO, wherein DMVR is performed on thenon-updated motion information for the current block to generate theupdated motion information, and the updated motion vector of the updatedmotion information is input as the searching start-point of BIO.

5.6. The method of any one of examples 5.1 to 5.5, wherein updating thenon-updated motion information based on multiple decoder-side motionvector derivation (DMVD) methods to generate updated motion informationfor the current block further includes: deriving multiple sets ofupdated motion information by the multiple DMVD methods, generating afinal set of updated motion information from the multiple sets of motioninformation.

5.7. The method of example 5.6, wherein generating the final set ofupdated motion information from the multiple sets of motion informationfurther includes: generating the final set of updated motion informationbased on an average or a weighted average of the multiple sets of motioninformation.

5.8. The method of example 5.6, wherein generating the final set ofupdated motion information from the multiple sets of motion informationfurther includes: generating the final set of updated motion informationby filtering the multiple sets of motion information using a medianfilter.

5.9. The method of example 5.6, wherein generating the final set ofupdated motion information from the multiple sets of motion informationfurther includes: assigning different priorities to the multiple DMVDmethods, selecting a set of updated motion information derived by theDMVD method with the highest priority as the final set of updated motioninformation.

5.10. The method of example 5.9, wherein the decoder-side motion vectorrefinement (DMVR) is assigned the highest priority.

5.11. The method of any one of examples 5.1 to 5.5, wherein performing,based on the updated motion information, a conversion between thecurrent block and a bitstream representation of a video including thecurrent block further includes: performing motion compensation usingmultiple sets of updated motion information derived by the multiple DMVDmethod, respectively, to obtain multiple sets of motion compensationresults, generating the current block based on an average or a weightedaverage of the multiple sets of motion compensation results.

5.12. The method of any one of examples 5.1 to 5.5, wherein performing,based on the updated motion information, a conversion between thecurrent block and a bitstream representation of a video including thecurrent block further includes: performing motion compensation usingmultiple sets of updated motion information derived by the multiple DMVDmethod, respectively, to obtain multiple sets of motion compensationresults, generating the current block by filtering the multiple sets ofmotion compensation results using a median filter.

5.13. The method of any one of examples 5.1 to 5.5, wherein updating thenon-updated motion information based on multiple decoder-side motionvector derivation (DMVD) methods to generate updated motion informationfor the current block further includes:

assigning different priorities to the multiple DMVD methods, selectingDMVD methods with the highest N priorities and being valid from themultiple DMVD methods, N being an integer and N>=1, generating, for thecurrent block, updated motion information based on the N DMVD methods.

5.14. The method of any one of examples 5.1 to 5.13, wherein the currentblock is a prediction unit.

5.15. The method of any one of examples 5.1 to 5.14, wherein thenon-updated motion information includes non-updated motion vector andreference picture for each prediction direction.

5.16. The method of anyone of examples 5.1-5.15, wherein the method isnot applied in case that the current block meets a certain condition.

5.17. The method of example 5.16, wherein the certain conditionspecifies at least one of: a size of the current block, a slice type ofthe current block, a picture type of the current block and a tile typeof the current block.

5.18. The method of example 5.16, wherein the certain conditionspecifies that a sample number that the current block contains issmaller than a first threshold.

5.19. The method of example 5.16, wherein the certain conditionspecifies that minimum size of a width and a height of the current blockis smaller than or no larger than a second threshold.

5.20. The method of example 5.16, wherein the certain conditionspecifies that a width of the current block is smaller than or no largerthan a third threshold, and/or a height of the current block is smallerthan or no larger than a fourth threshold.

5.21. The method of example 5.16, wherein the certain conditionspecifies that a width of the current block is larger than or no smallerthan a third threshold, and/or a height of the current block is largerthan or no smaller than a fourth threshold.

5.22. The method of example 5.16, wherein the method is applied in asub-block level, in case that a width and/or a height of a block, towhich the sub-block belongs, is equal to or larger than a fifththreshold.

5.23. The method of example 5.22, wherein the current block is splitinto multiple sub-blocks and each of the multiple sub-blocks furtherundergoes a bidirectional optical flow (BIO) process, in a same way as anormal coding block with a size equal to the sub-block size.

5.24. The method of anyone of examples 5.18-5.22, wherein each of thefirst to fifth thresholds is pre-defined or signaled in sequenceparameter set (SPS) level, or picture parameter set (PPS) level, orpicture level, or slice level, or tile level.

5.25. The method of example 5.24, wherein each of the first to fifththresholds is defined depending on coded information including at leastone of a block size, a picture type, and a temporal layer index.

5.26. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement the method in any one of examples 5.1 to 5.25.

5.27. A computer program product stored on a non-transitory computerreadable media, the computer program product including program code forcarrying out the method in any one of examples 5.1 to 5.25.

From the foregoing, it will be appreciated that specific embodiments ofthe presently disclosed technology have been described herein forpurposes of illustration, but that various modifications may be madewithout deviating from the scope of the invention. Accordingly, thepresently disclosed technology is not limited except as by the appendedclaims.

Implementations of the subject matter and the functional operationsdescribed in this patent document can be implemented in various systems,digital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.Implementations of the subject matter described in this specificationcan be implemented as one or more computer program products, i.e., oneor more modules of computer program instructions encoded on a tangibleand non-transitory computer readable medium for execution by, or tocontrol the operation of, data processing apparatus. The computerreadable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing unit” or “dataprocessing apparatus” encompasses all apparatus, devices, and machinesfor processing data, including by way of example a programmableprocessor, a computer, or multiple processors or computers. Theapparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of nonvolatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

It is intended that the specification, together with the drawings, beconsidered exemplary only, where exemplary means an example. As usedherein, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. Additionally, the use of “or” is intended to include“and/or”, unless the context clearly indicates otherwise.

While this patent document contains many specifics, these should not beconstrued as limitations on the scope of any invention or of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of particular inventions. Certain features thatare described in this patent document in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

What is claimed is:
 1. A method for video processing, comprising:determining original motion information associated with a current block;generating updated motion information based on specific prediction mode;and performing, based on the updated motion information, a conversionbetween the current block and a bitstream representation of a video dataincluding the current block, wherein the specific prediction modeincludes one or more of bi-directional optical flow (BIO) refinement, adecoder-side motion vector refinement (DMVR), frame-rate up conversion(FRUC) techniques or a template matching technique, wherein the updatedmotion information comprises updated motion vectors and the updatedmotion information is stored for some sub-blocks and for the otherremaining sub-blocks, the non-updated motion information is storedwithin one block.
 2. The method of claim 1, wherein the updated motionvectors are used for one of more of motion prediction for codingsubsequent video blocks, filtering or overlapped block motioncompensation (OBMC), motion prediction in advanced motion vectorprediction (AMVP) mode, merge mode and affine mode, and wherein thefiltering includes deblocking filtering.
 3. The method of claim 1,wherein the updated motion information is used for a first module andoriginal motion information are used for a second module, and the firstmodule is motion prediction module, and the second module is deblockingmodule.
 4. The method of claim 1, wherein the motion prediction is usedin processing blocks after the current block in a current picture orslice, or the motion prediction is used in processing pictures or slicesto be processed after a current picture or slice including the currentblock.
 5. The method of claim 1, wherein the updated motion vectors areonly used in motion information prediction of coding units (CUs) orprediction units (Pus) that are not immediately following the currentblock in processing order, the updated motion vectors are not used inmotion prediction of CUs/PUs that are next to the current block inprocessing order, and the updated motion vectors are only used aspredictors for processing subsequent pictures/tiles/slices, and theupdated motion vectors are used as temporal motion vector prediction(TMVP) in advanced motion vector prediction (AMVP) mode, merge mode oraffine mode and the updated motion vectors are only used as predictorsfor processing subsequent pictures/tiles/slices in alternative temporalmotion vector prediction (ATMVP) mode and/or spatial-temporal motionvector prediction (STMVP) mode .
 6. The method of claim 1, whereininformation including whether to update MV for BIO coded blocks and/oruse the updated MV for motion prediction and/or how to use the updatedMV for motion prediction is signaled from an encoder to a decoder andthe information is signaled in Video Parameter Set (VPS), SequenceParameter Set (SPS), Picture Parameter Set (PPS), Slice header, codingtree units (CTUs) or CUs, and wherein the method further comprisingupdating motion information that includes updating motion vector andreference picture for each prediction direction at block level.
 7. Themethod of claim 1, wherein the updated motion vectors are only storedfor inner sub-blocks that are not at PU/CU/CTU boundary, and the updatedmotion vectors for inner sub-blocks are used for motion prediction,deblocking, or OBMC; or wherein the updated motion vectors are onlystored for boundary sub-blocks that are at PU/CU/CTU boundary.
 8. Themethod of claim 1, wherein updated motion information from a neighboringblock is not used if the neighboring block and the current block are notin a same CTU or a same region with a size of 64×64 or 32×32, and theneighboring block is marked as unavailable if the neighboring block andthe current block are not in the same CTU or the same region with a sizeof 64×64 or 32×32; and wherein non-updated motion information is used bythe current block if the neighboring block and the current block are notin the same CTU or the same region with a size of 64×64 or 32×32.
 9. Themethod of claim 1, wherein the updated motion vectors from a neighboringblock are not used if the neighboring block and the current block arenot in a same CTU row or a same row of regions with a size of 64×64 or32×32, and the neighboring block is marked as unavailable if theneighboring block and the current block are not in the same CTU row orthe same row of regions with a size of 64×64 or 32×32; and whereinnon-updated motion information from the neighboring block is used by thecurrent block if the neighboring block and the current block are not inthe same CTU row or the same row of regions with a size of 64×64 or32×32.
 10. The method of claim 1, wherein the motion information of ablock is not updated if a bottom-most row of the block is a bottom-mostrow of a CTU or a region with a size of 64×64 or 32×32, and the motioninformation of a block is not updated if a right-most column of theblock is a right-most column of a CTU or a region with a size of 64×64or 32×32.
 11. The method of claim 1, further comprising predictingmotion information of blocks/CUs within the current CTU based on theupdated motion information or the non-updated motion information ofneighboring CTUs or regions.
 12. The method of claim 11, wherein theupdated motion information from left CTU or left region are used for thecurrent CTU, and/or the updated motion information from top-left CTU ortop-left region are used for the current CTU, and/or the updated motioninformation from top CTU or top region are used for the current CTU,and/or the updated motion information from top-right CTU or top-rightregion are used for the current CTU.
 13. The method of claim 12, whereineach of the one or more regions have a size of 64×64 or 32×32.
 14. Themethod of claim 1, wherein the method is not applied in case that thecurrent block meets a certain condition.
 15. The method of claim 14,wherein the certain condition specifies at least one of: a size of thecurrent block, a slice type of the current block, a picture type of thecurrent block and a tile type of the current block, the certaincondition specifies that a sample number that the current block containsis smaller than a first threshold, the certain condition specifies thatminimum size of a width and a height of the current block is smallerthan or no larger than a second threshold, the certain conditionspecifies that a width of the current block is smaller than or no largerthan a third threshold, and/or a height of the current block is smallerthan or no larger than a fourth threshold, and the certain conditionspecifies that a width of the current block is larger than or no smallerthan a third threshold, and/or a height of the current block is largerthan or no smaller than a fourth threshold, and .
 16. The method ofclaim 14, wherein the method is applied in a sub-block level, in casethat a width and/or a height of a block, to which the sub-block belongs,is equal to or larger than a fifth threshold.
 17. The method of claim16, wherein the current block is split into multiple sub-blocks and eachof the multiple sub-blocks further undergoes a bidirectional opticalflow (BIO) process, in a same way as a normal coding block with a sizeequal to the sub-block size.
 18. The method of claim 15, wherein each ofthe first to fifth thresholds is pre-defined or signaled in sequenceparameter set (SPS) level, or picture parameter set (PPS) level, orpicture level, or slice level, or tile level and each of the first tofifth thresholds is defined depending on coded information including atleast one of a block size, a picture type, and a temporal layer index.19. An apparatus in a video system comprising a processor and anon-transitory memory with instructions thereon, wherein theinstructions upon execution by the processor, cause the processor toimplement a method for video processing, comprising: determiningoriginal motion information associated with a current block; generatingupdated motion information based on specific prediction mode; andperforming, based on the updated motion information, a conversionbetween the current block and a bitstream representation of a video dataincluding the current block, wherein the specific prediction modeincludes one or more of bi-directional optical flow (BIO) refinement, adecoder-side motion vector refinement (DMVR), frame-rate up conversion(FRUC) techniques or a template matching technique, wherein the updatedmotion information comprises updated motion vectors and the updatedmotion information is stored for some sub-blocks and for the otherremaining sub-blocks, the non-updated motion information is storedwithin one block.
 20. A non-transitory computer readable media storingcomputer program instructions for performing, when a computer executesthe computer program instructions, a method for video processing,comprising: determining original motion information associated with acurrent block; generating updated motion information based on specificprediction mode; and performing, based on the updated motioninformation, a conversion between the current block and a bitstreamrepresentation of a video data including the current block, wherein thespecific prediction mode includes one or more of bi-directional opticalflow (BIO) refinement, a decoder-side motion vector refinement (DMVR),frame-rate up conversion (FRUC) techniques or a template matchingtechnique, wherein the updated motion information comprises updatedmotion vectors and the updated motion information is stored for somesub-blocks and for the other remaining sub-blocks, the non-updatedmotion information is stored within one block.